The Rise (And Fall?) Of Real-Time Search

Blogging and micro-blogging represent a stream of real-time web activity; from important real-life events to random personal thoughts, more Internet users are putting what they see, hear, think, and feel into this stream. Twitter, in particular, is a river of information. Breaking news is often reported there first, ahead of any other media — online or not. Just last week, Danny Sullivan wondered why the major search engines don’t offer a search service that taps specifically into this stream of information.

Two other services are aiming to do just that, but Twitter itself could prevent real-time search on other platforms from having a chance of success.

Twingly offers what it believes is the first federated microblog search service: Twingly Microblog Search. The search engine includes Twitter, Jaiku (which Google is officially abandoning), Pownce (already shut down, but Twingly has six months of archives), Identi.ca, and several smaller microblog services from around the world.

Real-time search across multiple platforms sounds promising, but because Twitter is the 800-lb. gorilla in this space, a Twingly search will tend to look a lot like Twitter Search.

That’s not Twingly’s fault; it’s a reflection of Twitter’s dominance.

Meanwhile, does the name PubSub ring a bell? It launched years ago as a real-time RSS/Usenet/breaking news search tool, then shut down in 2007. ReadWriteWeb reports that PubSub is coming back and hopes to be a real-time blog search service.

ReadWriteWeb says PubSub, which is in a closed beta right now, should get updates “only seconds after a story” is published because it’s part of the Ping-o-Matic service. That puts it in competition with Google Blog Search, My Yahoo, Technorati, and other sites that get pinged when new blog posts go live. PubSub, the RWW article says, also accesses some Twitter feeds. But those updates are slower, presumably because PubSub doesn’t tap into the Twitter API.

The Twitter API allows external services to pull in what it calls a “Firehose” of activity on Twitter — all the posts as they happen, and more. Without using the API, it might be difficult, maybe impossible, for search engines to gather Twitter activity in a timely manner.

And there’s the fly in this soup: For all the effort to put value into real-time search — particularly where Twitter is concerned — Twitter itself may be putting an end to those efforts just as they’re getting started.

Jesse Stay, the founder of SocialToo, wrote yesterday that Twitter has announced new limits on its API that will take effect this week. Stay writes that will have a dramatic impact on the ability of other services to tap into the Twitter stream:

“… this is going to hurt every app out there. I’m arguing that 20,000, or any request-rate limit for that matter, limits any app out there from being able to develop on the Twitter platform, and I don’t see why any able-minded entrepreneur would want to build on it if there’s such a rate limit in place.”

If accessing the stream becomes more difficult, or limited, it may be that Twitter’s own search engine will become the only place real-time search has a chance to grow.