Twitter’s New SEO Play: A Profile Directory Of All User Accounts

twitter-new-logo-2012Twitter has quietly launched a user profile directory that it hopes will bring it more search engine traffic.

It’s Twitter’s second SEO play in recent weeks, coming on the heels of a change in its robots.txt file. With that change, Twitter made its content-based search results — i.e., search results for tweets and hashtags, but not search results for users — crawlable.

With the new directory, Twitter is finishing that process by making its user profiles more discoverable by Google, Bing and other search engines. When asked why Twitter launched the user directory, a company spokesperson said it’s “to help people find the accounts they’re looking for with various search engines.”

Is The Directory Helping Twitter’s Search Visibility?

Twitter’s user directory has been online for a couple weeks. It’s essentially a master site map of Twitter accounts listed alphabetically and, from an SEO perspective, follows best practices by showing no more than 100 profile links per page.

As I write this, a Google search for gives a superficial display of 718,000 matching results. If I click through to reach the end of the 1,000 results that Google will show, I can reach page 47 before getting the “we have omitted some entries very similar to the 469 already displayed” message. And if I click to include the omitted results, I hit the end at page 57.


The 718,000 results isn’t right because Twitter’s directory doesn’t have that many pages. If you estimate that Twitter has about five million total accounts, the directory — with 100 accounts per page — would be about 50,000 pages, far less than 718,000.

UPDATE: I incorrectly typed and used five million accounts in the paragraph above, when I should’ve used 500 million accounts. My apologies. Based on 500 million accounts, the Twitter directory would have as many as five million pages, which means Google’s count of 718,000 could be an accurate reflection of how many pages it knows about.

But even if Google has only indexed about 57 pages (and those numbers aren’t perfectly reliable since I could be hitting a certain datacenter that has fewer results), it’s still a lot more than Bing. My searches show that Bing has so far only indexed the Twitter directory home page.


I have to assume that Bing will eventually crawl much deeper into the directory; Bing has gone on record admitting that its bot is slow, and that’s certainly the case here.

Of course, many Twitter profile pages have been crawled and indexed by Google and Bing for a long time now, so it’s difficult to say if the new directory is helping (or will help) the search engines find new, unindexed profiles.

The Checkered Twitter/Google Search History

What’s interesting is how Twitter’s recent search- and SEO-friendly moves fit into its checkered relationship with Google.

Google and Twitter used to be on good terms. Google had access to the full “firehose” of tweets and was primarily using that to power its Google Realtime Search product. But that ended in July 2011, forcing Realtime Search offline. Google promised that it would resurrect Realtime Search, but that never happened.

The end of that agreement meant Google no longer had access to the firehose — a special API feed of tweets — but Google was still able to do its regular crawl of Twitter profiles and content.

Except there was one temporary problem: Around the same time, in the summer of 2011, Twitter wasn’t very SEO-friendly. As we reported, Twitter was having technical infrastructure problems that hindered Google’s ability to crawl and index its site. Whoops.

In fact, Google says that it was cut entirely off from Twitter for a time. Here’s the head of Google’s search efforts Amit Singhal talking about the issue (at 1:35 in) earlier this year at our SMX London conference, as well as the challenge in crawling Twitter without a deal:

YouTube Preview Image

Twitter eventually fixed those issues, but its relationship with Google has been strained ever since the end of the companies’ agreement. In January, Twitter came out firing when Google launched its Search Plus Your World product, accusing Google of favoring its own social content ahead of Twitter (and other social content).

Google’s response then — and many times since — has been to mention how Twitter turned off the firehose and isn’t giving Google direct access to its content anymore. But Twitter engineers joined with peers at Facebook and MySpace to create a “Don’t Be Evil” tool that showed how easily Google could surface Twitter (and other social) content without firehose access. For more on that, see this recap of Danny Sullivan’s conversation with Google’s Eric Schmidt from January.

So today, while the war of words between Google and Twitter has died down, we now see Twitter taking a much more proactive approach to search engines — first with the robots.txt change and now with the SEO-friendly user directory.

For more on the directory itself and how it works, see my companion article on Marketing Land, Twitter Quietly Launches A User Directory.

Related Topics: Channel: Social | Features: Analysis | Top News | Twitter | Twitter: Search


About The Author: is Editor-In-Chief of Search Engine Land. His news career includes time spent in TV, radio, and print journalism. His web career continues to include a small number of SEO and social media consulting clients, as well as regular speaking engagements at marketing events around the U.S. He recently launched a site dedicated to Google Glass called Glass Almanac and also blogs at Small Business Search Marketing. Matt can be found on Twitter at @MattMcGee and/or on Google Plus. You can read Matt's disclosures on his personal blog.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Lukas John

    Never imagined Google would cheat that much. Pretends to have 632,000 results but I only get to page 57 (= 570 results)

  • Jean-Christophe G.

    Twitter has a lot more than 5M accounts.

  • Jaan Kanellis

    Why would Google only know about 700k profiles? Why not crawl and show millions?


    Sounds so good.

  • Matt McGee

    Yes, thanks Jean-Christophe. I mistakenly got the five million number stuck in my head while writing. It’s fixed now.

  • Matt McGee

    This is how Google has always shown its search results. It may tell you that there are 43 million matches for “widgets,” but it will only let you browse through the top 1,000 or less. Considering that almost no one goes beyond page one, it’s really not a big deal.

  • Magic Cube

    That’s a good progress for twitter. Anyway, you
    also might be interested with the new amazing Celluon Magic Cube, Bluetooth keyboard
    and mouse that provides an ergonomic way of typing on small devices such as
    iPads, iPhones, Tablets, smartphones and more. For more info, please visit a cool and ultra-portable, full-sized virtual
    computer keyboard.

  • Adult SEO

    Interesting development. Hopefully that means Google will rank relevant profile pages higher than individual tweets, as I see happening a lot right now.

  • Mukesh Chandra

    Its great news now individual profile also rank higher in google search

  • Matt McGee

    Do you have evidence of that, Mukesh? I’d be surprised if this user directory was helping anyone’s profile rank higher.

  • Clayburn Griffin

    Still no canonical URLs, though?

  • George Baily

    I think there is quite a difference to the number of pages which are crawled and silently indexed, and what will show up on a site: query on the public search engine. It’s a bit SEO naive to assume they should be the same thing.

  • Wouter Kiel

    Not sure why they’d launch it this way. There is zero added value for users, and these thin directory pages aren’t exactly SEO goldmines. And besides, what do they expect these pages will rank for? And if they do, what are users supposed to do on these pages if they land there?

    Why not stick with a good search box and that XML sitemap?

  • Anita Clark

    It is a step forward but it will be a very long time, if ever, before they get all the accounts indexed.

  • Lukas John

    I know. But since I can’t even go to page 100 means they don’t have more than 570 results.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide