Twitter itself has underscored the challenge as far back as March 2009 when it began showing a Suggested Users list to new users. Just a couple months ago, Twitter expanded its suggested users into about 20 categories and made those suggestions available to all users, not just new signups.

Still, whether you’re looking for friends, strangers or companies to follow, it’s never been an easy task. Beyond Twitter’s suggestions, there are search engines, directories, tools, and lists that can help. Here are some of our favorite ways to find the right people to follow on Twitter.

Twitter Lists

Lists are my preferred way to find people to follow. If someone I follow is on a list called, say, “Google Employees,” I can browse the other Twitter users on that list and add them if they look interesting. But that’s the slow, manual way to use lists, and there are a number of tools that make it even easier. I mentioned Twitter’s Suggested Users earlier — it’s the official recommendations divided into about 20 categories.

But you might also try Listorious, a search engine for Twitter Lists. If you’re looking for funny people to follow, maybe to make your morning workout more enjoyable, you can search for [comedy] and get back dozens of Twitter lists related to comedy. If you’re looking to connect with news reporters on Twitter, you can browse the [journalists] tag to see lists of reporters that other Twitter users have created.

Those aren’t your only list-based options. Twibes lets users build topical lists of Twitter users, and then (if I understand correctly) only shows tweets from those users that are specific to the topic. So, if I’m in the “SEO twibe” but send out a tweet about the Seattle Seahawks, it won’t show up in the SEO twibes stream.

Twitter Directories

Wefollow is a combination directory and search engine that you can browse or search by location or topics. The home page shows the most popular tags and cities, but there’s a search form if those aren’t what you’re looking for. Let’s say you’re looking for Twitter users in the tech industry to follow. The [tech] tag shows Twitter users under two tabs — Most Influential and Most Followers.

The Most Folllowers tab is self-explanatory; Most Influential appears to not only reflect follower counts, but also things like how often a user is retweeted.

There are a number of additional Twitter directories that might be good places to find Twitter accounts worth following. Try these:

• Twellow, which bills itself as the “Twitter Yellow Pages”
• ExecTweets, a directory of business executives on Twitter
• TweetFind
• Twitter Moms Network

Twitter Recommendation Engines

Mr. Tweet is a recommendation engine that relies on your contacts’ votes and also offers communities (like a directory) based on topical interests. It’ll show you “friends of friends” and tell you a little about the recommendations, like how many times your friends have retweeted them, how many followers and friends they have, and more.

Mr. Tweet also asks you to give public recommendations of the people you already follow and invites you to tweet about your influence as a Twitter user.

Here are several other Twitter recommendation engines that are worth checking out to find good users for following:

• Who Should I Follow? scans your current list of friends and makes recommendations of users you’re not following that are similar.
• Twiangulate is similar to Who Should I Follow?, but lets you discover who up to three other users follow in common. So, for example, you could use this to see who Danny Sullivan, Barry Schwartz, and Greg Sterling follow on a common basis.
• HiveMind does the same thing, but you can supply up to five users and it’ll tell you who they’re following in common.
• find2follow takes your Twitter username and gives you back a list of recommendations, though there’s no explanation how or why it chooses the accounts you see.

Twitter Search Engines

Tweepi is a new tool that lets you search and browse the followers of other users. Here’s a screenshot showing some of Barry Schwartz’s followers:

It includes a variety of stats about each user, like how often they tweet, how often they reply, how often they get retweeted, and so forth. In that sense, it’s one of the more advanced tools for searching Twitter users who might be worth following. (Thanks to @EricKing for the tip.)

Twitter’s Advanced Search tool offers a variety of ways to slice and dice users and their tweets, but the site also offers a few more specific user search options:

• Twitter’s Account Search helps you find people or companies you know are on Twitter.
• If you’re not sure if someone is already on Twitter, Twitter’s Friend Finder lets you locate contacts from Gmail, Yahoo, or AOL who have Twitter accounts.
• If you’re looking for someone who doesn’t have an account, you can invite them via email.

TweepML is a tool that lets you create and share lists of Twitter users, but it also offers a search engine and directory of popular lists, too. @JaredHuber says he does hashtag searches to find users worth following, then he uses TweepML to bulk-follow them.

Finally, HashTweeps lets you search Twitter users by how often they use hashtags.

Local Twitter Users

There are several tools available that make it easy to find Twitter users in your local area (or in any local area).

LocaFollow is probably the most powerful of these local Twitter search engines. You can search four ways at the same time: location, bio, name, and tweet content (i.e., keywords). So, if you’re specifically looking for a doctor in Seattle who’s on Twitter, LocaFollow can give you back a list that matches the “Bio” and “Location” field of Twitter users.

Once you get a set of search results, LocaFollow shows a variety of information about each Twitter user — how long they’ve been a member, their most recent tweet, and their following, follower, and tweet counts. If you sign in via your Twitter account, you can also follow users right from the LocaFollow interface.

ChirpCity and Nearby Tweets are two additional local Twitter search engines. Both search tweets in a given area, and both have additional keyword search options — so you can find people tweeting about “cars” or “movies” in your hometown, for example.

Twitter’s Advanced Search also lets you search for tweets in a given location, and you can combine that with other options like keyword search, usernames, and so forth.

All of these can be powerful tools for a small/local business that’s looking to connect with Twitter users in its hometown.

Final Thoughts

While it may be difficult to find the right users to follow on Twitter, the good thing is that there’s no shortage of tools and web sites that aim to make it easier. I’m sure there are even more than what’s listed above, so if you have a favorite tool, site, or method for finding Twitter users to follow, let us know in the comments — tell us what it is and how you use it most effectively.

I’m here at SXSW waiting for the Can the Real-Time Web Be Realized? panel to begin. I’ll be liveblogging what happens, so stay tuned.

Search engines have certainly jumped to add in ways to find real-time content. This panel features reps from three of them talking: Google, Microsoft and Collecta. Plus, you’ve got real-time guru Marshall Kirkpatrick in the mix. It won’t all be search, I’m sure, but it’s going to come up.

Our speakers, pictured above, from left to right:

• Marshall Kirkpatrick, ReadWriteWeb
• Jack Moffitt, Collecta
• Brett Slatkin, Google
• Scott Raymond, Gowalla
• Dare Obasanjo, Microsoft

Marshall to panel: What are some of your favorite use case of real time?

Jack: Communication has changed a lot from postal mail to telephone. To internet, closer to zero. More yells from the audience that people can’t hear. We’re getting more real time feedback about the session, heh. Giving latency down to zero gives new applications like instant messaging, Gowalla.

Brett: Future use cases, data interchange is interesting. Amazon has great inventory system to track all they have. For many people, especially in SF, painful to buy because you don’t support your local community. So a barcode scanner that did a feed that connected to local supply chain? You could get scale better than large corporations can do on their own and help local businesses compete better. That’s one of the promises of real time web, turning competition around. Smaller players can compete better.

Scott: Top of his mind are trending spots, tying in geodata with real time web. Where’s the hotspot right now? Then the tide will shift, so how do you track it. [This is what Gowalla does, track hot areas, I guess -- I need to try it. Can't Foursquare and Plancast and Gowalla just all get together?]

Dare: One way is make web pages real time, interesting if you can make parts of pages real time. Good example, hash tag for this conference. Type that in to a search, you get a stream but not pages. Getting more and more pages doing this rather than people would help. And now hard to hear him as we get bad speaker feedback. And I’m like in second row from front. Echo, echo, echo. It’s real time echo, and I can’t hear. Being able to actually analyze the flow of information is also interesting. Lots of patterns about good movies, friends.

Dammit, I wanted more c0ncrete examples. Nice try with the question, though, Marshall.

Marshall: One of his favorite things is something Brett said, that the system is being designed to support unplanned use cases (which if you think about that, is pretty awesome if it really works, how do you plan for things you don’t know?).

Jack: Lucky that first problem is solved, that publishers will give them the data. Second problem, still wrestling with, to figure out what the data formats for real time info will be. If you want to enable the real time web, need a way to push to say there’s data here, so come get it. PubSubHubBubGlub that no one can pronounce is one example of this. It so needs a better name — that’s me thinking that, but I’m not alone.

Scott: Right now, you can send messages from Buzz to Facebook, but you can’t replicate data for easily. I guess share at once to all of them. It’s really, really hard to hear, sorry. I guess everyone in the back can hear now. Maybe I can lip-read. What about corporate data policies that prevent integration from happening, and is it in the best interest of users. Or business deals that give some but not all of info. IE: Facebook, Twitter, Google, the data’s not yours to buy and sell for just your own benefit.

Dare: Common pain we all have today. Too long to get updates to Buzz from other services and so on. We hve to go about getting those services.

Brett: Standards are an issue. If you want to put monetization or business policies into a feed, you’re messing things up. Think that’s what he was saying, sorry. I’ve said it’s hard to hear, right?

Marshall: How about privacy?

Scott: We would love it if everyone was completely open with the world, makes things more valuable. But the user perspective, there’s marginal value in providing all your data. But it’s probably better to stay selfish with your own data. So trick is how to you find the right encouragement to help the business but let users stay in control. There’s a whole lot of work to be done but hasn’t been solved yet. Gowalla trying to give some incentives, like with gaming, so have to be willing to share to win.

Jack: Did we have these issues when phones came out. They decided to be public about numbers (actually, didn’t we all have to be public by default, because of party lines?). Lots of firehoses of data from MySpace or Twitter or Flickr. The level of support these streams have for deleting a post is bad. If you yank something off Flickr, there’s nothing in the Flickr data stream that tells others to delete this post. MySpace has some of this built in.

Brett: Lots of work to do on what do you share, how do you share. Then by addressing these problems together. The “oh shit” problem, oh I didn’t want to post that.

Marshall: Haven’t users already figured out where they want their real time stuff to be private, if they choose what to share or not at places like Facebook?

Jack: MySpace is a really different site than Facebook. In Facebook, you don’t have as much control over the user experience. Dare: If everyone on Twitter had a private feed, there would be no business model. So Twitter’s interest for people to be less private.

Scott: Similar to Amazon, they’ve never asked if they can analyze my data, but they just do it.

Question From Audience: Lots of Twitter and Facebook bashing. What are your companies doing right?

Marshall: Rephrases, what are your specific examples of good use cases?

Brett: Favorite problem is what’s on Google Buzz right now. Thing doing, encouraging competition in the community. Until now, people like Facebook have owned your data. Buzz is encouraging competition, you can pull in feeds from other sites.  Make a rich stream. Then expose it to the world without you knowing. No! He didn’t say that last sentence, you all just thought it. How do you deal with the challenge of different user groups. Today on Google Buzz you can link up sites to your profile (when it works, it’s still pretty lame with me), then next step back, if you host your own blog and that got pulled into Buzz with comments, then my comments there on Buzz, can it go back to your blog. Does that help people regain ownership they may feel about their content, get their comments back (maybe, but kind of good luck with that. it’s like saying you want to pull all the real world non web discussion back into your own living room). We don’t want to own anyone. At end of the day, competition will be the best for all of us.

Scott: Love to see more sites think about what’s the first 10 seconds of users experience.

Dare: Bing Twitter which is interesting way to zoom in on a location and see what’s being said (I think that’s Twitter on Bing Maps). A lot of work went into trying to find trending topics. Other thing interesting at Microsoft, Windows Mobile 7. He has four apps he jumps back and forth to find out what’s going on. I go to Fourquare to Twitter to Facebook. The thinking is that’s its not about going to 5 7 or 10 apps but as a person, I know Rob — here’s what Rob’s doing. Moving away from notion of status update and have people who are your friend get info to you.

Brett: But with Buzz, a criticism was we show ads. So how do you monetize that. Something like people Buzz, if you break down the barriers, lose the monetization for the brands that build the feeds, who’s going to do this. Specifically in the case of ?peoplehub peoplebuzz? dammit it’s so hard to hear.

Dare: I like to start from what’s the best experience for the customer. Doesn’t mean you have to lose the monetization. [Now moved to sit directly in front of speaker, maybe I can hear better that way]

Jack: Something like Twitter is doing, charging for their content, is that more of what you mean.

Dare: There’s no one size fits all. Some companies feel if you get content out as much as possible, that drives content. Hitwise stats show Facebook drives Foursquare the most traffic. Should they pull out unless paid? I don’t think so.

Audience member asks question on privacy:

Scott: We provided some basic degrees of control. If you go more fine grained, it becomes a user experience challenge. It’s a design problem more than anything. We’re working on it, and I definitely believe it is solvable, but I think the trick is to always be explicit with the user and show them where it is going.

Jack: I don’t think users have figured this out. Has friend with 12 year old daughter who had party at house, got busted, forgot her dad follows her. Write about your teacher, but it gets back to her. People have to figure out the potential repercussions, and I don’t think we’ve even figured these out. When our children have these experiences each day, then we can figure out the technology maybe [or it's not a tech issue but maybe teaching kids you don't share everything].

Brett: Not just tech issue. Part of what doing to set your preferences overall. Webfinger designed to set common privacy sharing, so you don’t have to remember and say how you want sharing to work everywhere [yeah, good luck with that, when like at Facebook, there's so much you shared in a granular fashion]. Every provider has different settings and models, and that’s wrong. The user should just say this is what’s important to me.

Jack: I don’t think users will understand the ramifications of those decisions. If I say it’s OK to share my Flickr photos to the world, there are still the ramifications.

Brett: But that’s because your privacy settings are inconsistent. That’ll change. So you get a job, you change those settings. We need consistency first.

Scott: Skeptical, with fine grained issues, touchy feely issues, so many shades as gray. I think it will be as successful as to have cross site identity, like OpenID, which has failed.

Dare: Same, hard to be consistent, that type of solution won’t work. Facebook model is different than Twitter follower model, how do you describe all these things that are different. What I tend to suspect are two things. One, users want privacy controls. Wehn you ask them, they have some expectations. But if you do introduce those, it produces friction. People don’t go into the sharing box and say this tweet is NSFW, so I will share it with only a few people. They just don’t share it. Even if you give them the functionality, it’s too much work. They don’t do it.

Another audience question, didn’t catch it, will dive into answers.

Jack: When you have a ton of info, like this blog post appears first, then this came second, you can dedupe and fight some spam. The real descrepancy is betweent different sites. Each Twitter ID is unique, you can tell. But if you put it on Facebook, harder. And Facebook has internal duplication because not all things open [think that's what he said].

Brett: Tech solutions trying to do. If Buzz hits Twitter, then Twitter hits Buzz, its gets worse and worse over time. Algos help, but not easy for little guys to do. Using Atom source [think he said] as original content producer can help. Also cross post extenstion from Six Apart to link feeds that are similar but still slightly different, to tell if you posted in multiple places.

Jack: Still going to be a hard thing no matter what. We have to make sure people actually do things.

Marshall: Thanks for joining us!

Like it main site, Collecta mobile includes real-time news, photos, and status updates from more than 10 million content sources — from Twitter to Flickr and CNN and blogs. The mobile site is available on the iPhone, Droid, Palm and Nokia Maemo. As you can see from the screenshots (below) that Collecta provided, the company is hoping to take advantage of the interest in real-time news and information surrounding the upcoming SXSW conference.

The first group was told to search for a product they might buy, and were called the “consumer” group. The second group was told to simply look for information on a product, and were called the “forager” group.

The study, conducted last month, sought to answer the following questions:

1. Does the average internet user recognize and understand real-time results?
2. Are consumers finding and clicking on real-time results?
3. And simply, the bird’s eye view: what are the consumers saying about real-time results?

At the highest level the results showed the following:

• 73% had never heard of real-time results before participating this study
• Only a quarter of the consumers cared for the real-time results compared to 47% of the information foragers
• The majority of the participants surveyed were indifferent to the real-time results

Here’s the age distribution of the 44 participants:

Below are screens showing the differences in the eye movements of the two groups. Remember the only difference between the two groups was nature of the task assigned. The “real time results” are called out in a box at the bottom in this set of examples:

The study, which contains much more discussion than my short summary above, seems to assert that real-time results so far have limited impact or appeal. While there’s enormous hype and coverage among tech-insiders about “real time search,” the public doesn’t really understand (or perhaps care) what the fuss is all about.

The widget builder is available now at widget.collecta.com, and includes several customization options including widget title, header background image, and even use of an external CSS file. It appears that the size of the widget can be adjusted by editing the code created by the widget builder. The builder creates a short piece of iframe-based code for embedding; WordPress bloggers can also use a plugin to add the widget to their blogs.

You can see the widget in action on the MySpace Breaking News page, or just have a look at this sample LOST widget screenshot.

(image provided by Factery Labs)

On the screenshot above (you can click for a larger version on Flickr), Factery Labs is showing hot topics like “california storms” and “supreme court ruling.” But rather than showing important tweets, tweets from important people, or the most popular links for each topic, it’s showing me the facts related to them. Under the “supreme court ruling” tab on the right, for example, a quick glance tells me that a 5-4 Supreme Court vote “loosened restrictions on corporate campaign spending and had President Barack Obama fuming.” It seems to me that this is exactly what real-time search needs: information, depth, and context.

During a demo call last week, Factery co-founders Paul Pedersen, who previously worked at Google and Infoseek, and Sean Gaddis, formerly at Netscape, eBay, and Skype, explained where the real-time facts are coming from and what happens behind the scenes. (Not on the call but worth mentioning here is that Factery’s lead engineer Nitay Joffe has spent time at both Google and Powerset.)

How It Works

Factery Labs tracks the trending topics from the Twitter and Google Trends APIs to determine what’s hot right now. It identifies relevant tweets and URLs from Twitter, and also taps into Yahoo BOSS to identify additional URLs that may be relevant to the trending topics. Once it has this collection of URLs, it scans them for facts and applies its FactRank technology (a nod to Google’s PageRank) to build what users see on the site’s “Trends” tab.

“We drill past the links and get into the pages to see if there’s something valuable to the user’s request,” Pedersen said last week. “It’s not news; it’s news according to what the web says is news. We show what’s happening as defined by the web, not by an editor.”

There are seven other tabs, covering topics like sports, politics, entertainment, and tech. The topics that show up on these are user-defined, and are not necessarily trending topics from Twitter and Google Trends.

Users can conduct searches and “favorite” their search by clicking a yellow star; this adds the query to the main panel and saves it there for when the user returns to the site. Each fact can also be shared via Facebook, Twitter, or email.

Strengths & Weaknesses

Factery’s fact engine is, to me, a compelling tool for anyone who’s interested in current news and events. The focus on facts over recent/hot tweets and popular links eliminates a lot of the noise that’s sometimes associated with trending topics and real-time search. It’s like Google News, only without all the blue links and the need to click five articles to find out what’s going on.

But there’s still room for improvement. It’s not completely free of noise/spam — I saw ticket agency offers appearing in the results for some sports-related topics. It sometimes pulls odd facts for topics on the non-trending tabs, like a legal disclaimer from NBA.com (see image at right). There are also some API issues with displaying the source of the facts; rather than seeing the real source of the fact, you’ll occasionally see Bit.ly and other shortened URLs. Factery says it’s working on a fix for that.

Conclusion

What I like about Factery’s fact engine is that it does the grunt work for me. It not only shows me what’s hot in the news right now, but it also tells me why. Popular tweets and links are part of the real-time equation, but some of that is noise; I think there’s real value in Factery’s focus on the facts.

And I can’t help but wonder, as Google and the other big search engines continue to emphasize answers, facts, and information “shortcuts” in their search results … is this a preview of what all search engines will look like someday?

Just a couple weeks ago, OneRiot announced that it’s indexing and including MySpace content in its search results. Google has also said it’ll include MySpace updates in its real-time search results, but it’s not there now.

Both Google and Bing have coveted the “status updates” that users make on social sites like Twitter and Facebook. Our What Is Real Time Search? Definitions & Players explains how status updates, also called microblogging, works at these places.

Updates often contain great content, such as information that doesn’t exist anywhere else on the web or which can be mined to see the popular links that are being shared. Having updates is essential for anyone who wants to run a real time search service. And when it comes to status update content, Twitter and Facebook have more of it than anyone else.

On October 21, Bing announced a deal to get update data from both Twitter and Facebook. It also launched a Bing Twitter search engine that same day, promising Facebook information would be used in some way in the future. Also that same day, Google announced a deal with Twitter. For those keeping score, it was Bing 2, Google 1. Bing had both Twitter & Facebook, while Google was Twitter only — and Google was not even using that Twitter information yet.

Last week, on December 7, Google pulled even with Bing. It launched a Google real time search engine, that primarily focused around Twitter data. Google said it had also reached a deal with Facebook for data, apparently giving Google parity with what Bing had. That data would be integrated into Google’s new real time search in the future, the company said. You could even argue that Google pulled ahead of Bing, announcing it was also taking in status updates from MySpace. But really, it’s the big 2 that matter here: Twitter and Facebook. Bing and Google both had deals with them; both were “even.”

Yesterday, during a visit at Facebook, it dawned on me that this wasn’t the case. In discussing the arrangement, Facebook explained to me that Google was only getting information from Facebook fan pages. Personal updates weren’t being provided.

Fan pages? Personal updates? Consider this. I have a personal page on Facebook where I can share updates with friends, though I can also choose to share those updates with “Everyone” on Facebook. I also have a fan page where I can post updates. Many companies have fan pages, such as Coca-Cola.

Google is only getting information from fan pages. Those millions of people on Facebook putting out updates via personal pages? Google’s not getting any of that.

Did I just miss this from the Google press conference I attended about the deal? Yes. Google vice president of search products & user experience said:

Facebook will be providing us with a feed of updates from their public profile pages, also known as Facebook Pages.

Others caught this distinction, such as Inside Facebook and the Washington Post. But for those like me who may have missed the difference, I thought a further drill-down would be useful.

As for Bing, it will receive updates from fan pages, just like Google. But in addition, Bing will receive any updates done on any personal pages, as long as those are marked for sharing with “Everyone.” Facebook also said that Bing is the only service outside of Facebook itself that currently has access to this data.

Why isn’t Google getting personal updates? Google had no comment about that. Facebook told me:

The terms of the Google deal were specifically related to public updates on Pages. But our conversations with both companies are continually evolving regarding ways in which we might work together in the future.

As for the deal, Google was steadfast in refusing to confirm if there was any financial component involved. IE, was Google paying to get the Facebook feed? While Google wouldn’t say, Facebook would. No. As with Bing, Facebook is not charging Google for its updates. Facebook  told me it finds it worthwhile to provide an update stream because ultimately, it may bring people back to the Facebook site, where visitors then see ads.

Finally, Microsoft said it has no news to share on when Facebook updates would be integrated into Bing. Checking Google’s real time search service today, I can’t see that fan page updates have yet been integrated, either.

This week, Google rolled out real time search integrated into its main results. It’s a dynamic, scrolling display of the latest tweets and other content that matches what you’re searching for.

Google search results that are scrolling? Think about that. Four years ago, Google quite famously said there would be no “crazy, flashy, graphical doodads flying and popping up all over the Google site. Ever.” I don’t know about you, but a dynamic scrolling box of results is pretty flashy and graphical to me.

Google Revives Instant Add

More dramatic, Google is effectively going back to the old days of 1997, when Infoseek had an “instant add” feature that allowed anyone to add their pages to its search index within seconds. That was dropped, because it opened the door to search spam. People would just rewrite their pages until they got a top ranking.

Fast-forward 12 years to today, and if I want a top ranking on Google for Copenhagen, it’s easy. I just tweet something about it and moments later, I’m there for a search on copenhagen:

This is transitory, of course. I’m only going to be there for a few seconds. But spammers are smart. They’re happy to set up robotic accounts that do nothing but tweet all day. And Google’s smart, and can spot abuses like this. But Google’s not perfect, and spam will get through, and it’s an entire new cat-and-mouse game it’s playing.

For more on the spam potential, I highly recommend checking out Google Enables Real Time Spam and More from Outspoken Media and The anatomy of a deceptive Tweet spamming Google Real-Time Search from Sebastian’s Pamphlets.

Pros & Cons Of Real Time Integration

Why would Google expose itself to a new front in the spam wars like this? Is real time search THAT compelling? Sometimes, sure. When doing a search on Obama, it was interesting and compelling to see some reactions from those listening:

I didn’t have to go out of my way for these “person on the street” views. They were part of my overall search experience, and useful.

But then again, I didn’t need them scrolling. Google could have easily done a “snapshot” and shown me some static examples, with a link to get more. Perhaps it might also show me what it determines to be the most authoritative or most retweeted comments.

Scrolling isn’t necessary. It’s not like with the longstanding news results OneBox that shows that it scrolls. Yet some of that same content is now in the real time area where, by virtual of being sprinkled with real time pixie dust, scrolling is deemed important.

It’s News! It’s Real Time! It’s Real Time News! Or Something

Speaking of news, it gets mixed into the real time results — and that’s darn confusing. Here’s what I saw when I searched for obama nobel peace prize on Google:

I get news results, then underneath that, I pretty much got news results again — only with scrolling. The first arrow points to a story from Examiner.com, which was just featured in Time Magazine yesterday about how it has super powers to do whatever it wants within Google News. Apparently these extend to real time search, because I got three stories in a row from the publication, as the next arrows point at.

Of course, these stories weren’t each written within seconds of each other. The probably weren’t even published within seconds of each other. Google either just found them all at once or perhaps got pinged (and maybe repinged) that they were all published. But that’s the farce of mixing in things that aren’t microblogged content into real time results.

My What Is Real Time Search? Definitions & Players article explains the distinction more. To its credit, Google does provide a way to filter to just microblogged content, if you want that (my Google Launches Real Time Search article explains more about this).

In other instances, it’s hard to see how giving up so much of Google’s most important search real estate is enhancing my experience. On the trending topic of e-bay, according to Google, I got:

The latest eBay items for sale, as tweeted. I guess that’s good. But why not take a feed from eBay?

The Spam Challenge

To its credit, I watched for outright spam in a variety of ways for about 20 minutes and didn’t see any. For example, consider this tweet on Tiger Woods from Twittter Search:

That never made it into Google. But nevertheless, stuff will get through. Yesterday, Google was listing “woods tiger” as a trending topic, so I checked it out:

Whatever #chuckfest is, it was hoping to cash in on a variety of trending topics (Ustream, Tiger Woods, Christmas) by listing them along with its own hashtag.

If Your Friends All Jumped Off A Real Time Cliff…

Google’s not alone, of course. Today, Yahoo integrated Twitter into its results. Because? Well, because if Google has them, Yahoo better do it. And Google has them because? Well, Bing rolled out a Twitter search service, so Google had better get one.

Meanwhile over at Facebook, it’s doing extremely disturbing things with privacy settings that all seem in hopes of getting more people to effectively tweet out of Facebook itself. Facebook wants an update stream that rivals what Twitter has. Because? Because I guess Twitter’s got all this attention over its update stream, so Facebook better get its own going lest it misses out.

It doesn’t seem to matter that Twitter’s not yet making much money off its service. Facebook is willing to dive in and risk seriously upsetting existing users with privacy changes to chase the Twitter-rainbow.

Meanwhile, MySpace rolls out an API to better release its updates, all of which sounds as if it’s screaming, “Hey, we’ve got updates too!” And Google’s cut a deal to carry those, though it won’t say if it’s paying for that deal or not.

Trust me, I know. I asked Google on Monday three times, in three different ways, if they were paying for data from MySpace, Facebook or Twitter. Nada. They won’t say. Which leads to the craziest part of all this.

We’ll Pay For Chatter, But Not For News

Google is steadfast that it does not want to pay for news content. Sorry, Rupert Murdoch (who also MySpace), we don’t think we should pay to link to your news stories. And yet, Google does appear to be paying Twitter for its data.

The New York Times quoted Twitter CEO Evan Williams as saying there’s a financial component to the Google deal, though that wasn’t the “focus” of it. Twitter COO Dick Costolo confirmed the same at a TechCrunch event recently.

Facebook is less certain. The Daily Telegraph reported that Facebook COO Sheryl Sandberg saying that Facebook wasn’t making any money from its deal with Bing. Maybe the same is true for Google. Maybe not.

It sure is a weird strange world where it’s deemed crucial to do deals over what people are saying about news rather than the news itself. I also feel a little pimped, like what I do on Twitter, Facebook and MySpace has become their property to sell and trade. I know — I use these services freely, and I can lock things down if I want. Still, it feels odd in some way.

It also just feels weird that there’s so much effort to do something, anything with real time in terms of search rather than the right thing. Throw something on the wall now! See if it sticks.

Of course, I don’t know what the right thing is. We do need experimentation. We do need to have things thrown on the way. And I DO think real time results in some way belong in the main search results, as well as dedicated tools. Plus, I do believe that there’s a business model in Twitter, that microblogging is an important new communications channel.

Still, I also can’t help feeling the search players are all a little tipsy off that real time catnip.

How do you get it? It’s rolling out over the next few days, and when you have it, you’ll see it just happening withing your search results. But if you can’t see it, try clicking on this link which may force it to appear for you.

For example, below is a search for health care on Google where about midway down on the page, there’s a “Latest results” section that appears with real time results:

The results automatically flow with new information — and if that’s annoying, you can use the “Pause” link at the top of the section to stop them. What you can’t do is remove the section entirely, if you don’t like it.

By the way, you won’t always see the results. “If high quality information is coming in, then we will show it,” said Amit Singhal, a Google fellow who heads Google’s ranking systems and oversaw the development of the new real time search system, when I talked with him after the Google press conference.

In other words, if Google thinks something has some real-time component to it, then it will show the section. In particular, if Google sees a spike in information on a certain topic, along with queries on a particular topic, then it assumes there’s a real time situation happening — very simplified!

Click on the Latest results link, and you can drill down into the results:

What do you get? Singhal said:

• Tweets from Twitter
• Content from Google News
• Content from Google Blog Search
• Newly created web pages
• Freshly updated web pages
• FriendFeed update
• Jaiku updates
• Identi.ca updates
• TwitArmy updates

Down the line, content from Facebook and MySpace is promised — updates that come out of them and which users agree to share with the public. But right now, they aren’t there.

Google previously announced a deal with Twitter to take in its data. Bing has one too. Bing also has a Facebook deal for data, though it has yet to implement that data into search. Part of today’s news was that Google now has a Facebook deal plus one with MySpace, which Bing doesn’t have.

Google wouldn’t release any financial details about any of these deals. It wouldn’t even confirm if there ARE financial deals. For all we know, Google’s getting the information from Twitter, Facebook and MySpace for free. I believe Twitter’s suggested it is being paid from Google and Bing for its data (I’m checking on this). I believe Facebook has said there’s no financial arrangement between it and Bing for its data (again, checking on this).

MySpace is a particularly interesting situation. Rupert Murdoch’s News Corporation owns MySpace. Murdoch wants Google to pay for News Corp’s content. So when it comes to MySpace, is Murdoch getting paid for data there but not for Google News? Or is he happy to give that data away for free but not when it comes to news?

I haven’t had time to drill into the relevancy of what’s presented and compare it against places like Twitter Search or other real time meta services like Collecta.

How the information ranked? Singhal said only information deemed highly relevant is included. So spammy tweets, low quality pages and other content might not make it into the real time search “layer” that is used. After that, results are ranked by time.

How about ranking content in other ways. For example, if searching on a particular topic, does it make sense to show the most “authoritative” tweet first, rather than giving primacy to the latest ones?

“I would not be surprised if you start seeing far more of an emphasis on original tweets in our search,” Singhal said.

If you look at the second screenshot above, you’ll see that the “Latest” option in Google’s Search Options pane is flagged. Now look further up, and you’ll see that this is for “All Results” that Google has (or what’s called “Everything” results in the new Googel web search user interface being tested).

Under all results, there’s a new “Updates” option that can be toggled. So want to see the “latest” or real-time results that are only status updates (such as tweets?). Toggle the Updates option, and you’ll filter out other things like real-time news reports or new web pages that are added.

I like this feature, because to me, “real time search” means bringing back microblogged content, not news results, not freshly updated web pages and so on. For the detailed explanation about that, see my What Is Real Time Search? Definitions & Players post.

That post also explains a class of “real time” search that Google is not doing, such as allowing you to explore the most popular links that are being tweeted. The folks at some places like OneRiot can thus breathe a sigh of relief. Even Bing’s Twitter search retains some bragging rights.

Enter a URL into Google’s real time search service, and you don’t see much about what’s related to it in terms of real time commentary. For example, here’s what’s shown when I searched for information related to my live blogging of today’s Google news:

Do the same for Bing’s Twitter search, and you get much more content:

This may change, by the way. Google said that it be growing its real time search service over time, so watch and see. Also see Up Close With Bing’s Twitter Search Engine for more about Bing’s service.

Google also noted that it is expanding shortened URLs. You’ll still see the shortened version listed (say if it is from bit.ly or another URL shortener), but then you’ll also see the final destination URL. Content from that destination page is spidered.

Related to Google real time search is a new real times trends feature on Google Trends, which itself officially comes out of Google Labs today. Let’s do a screenshot:

On the right is what Google Trends had before, topics that are deemed popular based on searches entered into Google. By popular, these are things that are seen as out of the ordinary. Otherwise, you’d see queries like “sex” and “hotmail” always showing up.

On the left is what’s new, top trends based on Google’s analysis of real time content. This is important. The trends you see at Twitter, on its home page if you’re logged out, are based to my knowledge on what people are searching for on Twitter:

In contrast, Google’s real time trends — “Hot Topics” — are based on what it sees from analyzing the text of real time content.

I can’t say if one method is better than another. Both probably have advantages.

Below the Hot Topics list is a search box that effectively provides a front-end to Google real time search. Enter a search query, such as climategate, and you’ll get back matching real time results.

Currently, Google says there’s no plans for a dedicated standalone real time search page as you get for say Google News or Google Images. However, I can point you to a pseudo-page like this. Go here, and you’ll get a nice clean Google home page that is configured to search for real time information.

And how about real time search and its importance to Google overall? Was this really a major gap that could have killed the company?

Hardly, in my view. There’s no doubt that it’s good to have. It’s incredibly difficult to be a leading information source and yet when there’s an earthquake, people are instead turning to Twitter for confirmation faster than traditional news sources on Google can provide.

Google vice president of search products and user experience Marissa Mayer gave me a personal example of how having real time data can help Google. An avid skier, she found herself going to Twitter on a regular basis to get ski reports from people actually on the slopes, which she found more accurate than what some resorts and other reporting services were offering.

Google could have survived without a real time search component — in particular without the dedicated flow of microblogged updates — but it becomes a more complete and useful service with them. I’m glad to see the integration and am looking forward to see how it matures.

Still, in the long term for those trying to measure the search face-off between Google and Bing, it’s not real time search that’s the major battlefield. Instead, it’s personalized search that I think is far more important. And on Friday, Google unveiled with far less fanfare a major change in how it provides personalized results. For more on that, see:

• Google Now Personalizes Everyone’s Search Results
• Google’s Personalized Results: The “New Normal” That Deserves Extraordinary Attention

Finally, thinking you heard something previously about Google doing Twitter search? You did, but that wasn’t Twitter search. It wasn’t real time search, either. It was social search, where Google looks at those you follow through Twitter or other social networking services and then shows you content just from those people, in response to a search.

Our Google Social Search Launches, Gives Results From Your Trusted “Social Circle” article covers that system and how it works in more detail. It’s completely separate from real time search. In fact, potentially you could do a search and get a section with real time results on the same page as results from your social network.