• MattCutts

    Not to be a complete Twitter rube, but does Twitter provide the raw data such that it could be downloaded, indexed, and searched? I haven’t seen where one could do that, but certainly I could have missed it.

    P.S. We did increase the quote for that App Engine app yesterday. :)

  • http://searchengineland.com/ Danny Sullivan

    Google already crawls Twitter right now. I don’t see any reason why Google couldn’t offer a Twitter search, if it wanted to. Summize.com was doing this before Twitter bought Summize. The bottom line is that more and more, I’m turning to Twitter to get real time search results on a range of issues, stuff that neither Google Blog Search or Google News Search have, since they aren’t picking up Twitterstreams.

  • joehall

    This might be a lot slower than Google or Yahoo is used to, but they could always integrate twitter’s search data via simple XML output. This is what we do at WhosTalkin.com A down side to this strategy is that it takes longer to build an index, because it depends on the frequency of the tool’s use. It also may not be as dependent on relevancy as say Google’s algorithms because twitter will return the freshest content first which is not always the most relevant.

    But this brings up the question of why would a content based search engine that’s main focus is on relevant results want to develop a side service that is more dependent on a medium that is inherently social and not always content based? Especially when twitter already has a search capability that works fine.

    The Twitter BOSS mash up is a cool example for the purpose it serves but recreating twitter’s search, seems redundant.

  • MattCutts

    I did a little digging. It appears that the most recent blog post from Twitter about this is from October 2008 at http://dev.twitter.com/2008_10_01_archive.html , and the “firehose” of data (the entire stream of tweets) is not widely available yet:

    “We provided this stream on an experimental basis some months ago, but had to limit its distribution to just a few subscribers while we worked on technical hurdles. … I’m happy to say that we’ve staffed a project with three engineers (myself included) to find the best solution for distributing what we jokingly refer to as our “firehose” of tweets. We intend to have a solution as soon as possible, as we know that some of you have applications at the ready that depend on this stream.”

    My guess is that Twitter will tackle this more in 2009.

  • http://searchengineland.com/ Danny Sullivan

    Matt, sorry, the fact that Google doesn’t offer a search of twitters isn’t a Twitter issue. If Google thought providing this type of a service was a priority, you’d have been on it. You’d have been talking to Twitter. You’ve have at least played with doing something with the 20 million pages from Twitter you already index at the moment.

    Tiny Summize did this last year before Twitter bought them last year:
    http://blog.twitter.com/2008/07/finding-perfect-match.html

    Google wasn’t able to do what Summize was doing before they had a formal relationship with Twitter?

    I think the reality is that it’s not a Google priority to do this. I can understand that. It’s taken until only recently to recognize how valuable searching the twitterstream is. And it’s not like it’s a crisis or me or other searchers — I mean, we can use Twitter Search, after all.

    It’s just that this is a growing and key search area that Google’s not serving — and that’s your core product, search. I mean it’s cool in the sense that it does help reinforce the fact that Google doesn’t just dominate in all search areas, or is even trying to dominate in all search areas, as some believe you are.

  • joehall

    I still don’t understand why this should be a Google priority. I mean, I understand why Google would want to include twitter in its index, but why make a separate tool? Should Google make a separate tool for other popular social media sites? Like friendfeed?

  • digitalshaman

    Great observation!

    When there is enough value to justify (by some measure, eg, ROI) search it is hard to understand why Google &/or Yahoo are not more active … OTOH, I used to use Summize extensively. Since the purchase I find more value in TweetScan.

    Value in tweets can be observed in other ways (beyond freshness such as breaking news – I am in full agreement that Twitter leads in at least this area) – 1) manner of input/viewing given the posting limit 2) value in re-entering data that is objectively similar 3) re-entering such that it objectively differs 4) not enough value to re-post & 5) value in how, who & when another re-tweets the *original* post/tweet …

    Now, about the cost of TXT/SMS vis-a-vis other bandwidth is largely ignored though clearly there is growing leverage accruing to Twitter as it improves stability & opens itself to integration on other platforms (eg, Facebook, zenbe, etc.). Even if the Summize purchase may have been prematured (IMHO – tho I dont know the price).

    Would not be surprised if there are deals to split that “cost” (800-1200 bucks per MB) or moves to the freshness aspect of Twitter akin to push mail (eg, Blackberry) for which profitable business models can be sustained.

  • Philbradley

    Just as an observation – Icerocket is already offering a nice integrated Twitter option – either as a search tab, or as part of their Big Buzz integrated news option. Nice, simple and easy to use.

  • Philbradley

    @joehall I think it’s fairly clear as to why Google needs to be getting involved in this area. Because it’s search. I am increasingly using news resources to find the information that I’m interested in, and Google just isn’t cutting it any longer. The Google news option is only just about acceptable as a resource – I prefer to use engines such as Silobreaker which offer a much more rounded, comprehensive and integrated service.

    Now, when I want news, that’s what I’m after, and using social media resources is one of the key ways in which I do it. I don’t think it’s necessary for Google to create something new for every single one – that would be verging on the ridiculous, but some sort of integrated social search resource would at least put them in the ballpark if not on the field itself. After all, plenty of other search engines are already doing this.

  • http://thenoisychannel.com/ Daniel Tunkelang

    What I think we’re dancing around here is that “search” encompasses many different use cases.

    For Twitter, the main use cases seem to be following breaking news, monitoring the global conversation for mention of a particular topic, and people search. These use cases are quite different from what we normally associate with web search. Moreover, the first two are already adequately handles by Twitter’s existing search function (i.e., what they acquired via Summize). People search could be a lot better, but no one really does much there beyond known-item search. LinkedIn is probably the leader–but that may be more about their content than their technology.

    So, to answer the question of why Google and Yahoo don’t offer Twitter search, I think it’s because they’d be hard pressed to offer much beyond a wrapper around http://search.twitter.com/. The better question to ask is: what information seeking needs could a new, improved Twitter search engine better address?

  • vish makhijani

    Sorry I am late to this discussion but I think its worth calling out that we launched BOSS when I led the team at Yahoo because we wanted to expand the development platform for both external and internal developers.

    Unlike our friends at Google, we did not historically do a good job of unleashing developers across Yahoo to work on search. BOSS helps out dramatically in this regard. Folks in research, in hong kong, on the email team or wherever all have a shot at building interesting search products.

    Danny, clearly twitter search has promise but resources are tight at yahoo and now there is a platform that allows ideas to turn into products (instead of sitting in a long list waiting to get funding). The search team gets to see if they work and then feed them if they do.

    If external developers are successful using BOSS, one of two things happen (both are good).

    1) the search market grows (on yahoo’s platform) and hopefully ultimately gets monetized by yahoo’s advertising engine

    2) more competition grows for the market leader :-)

    The world needs clearly need to tap Twitter to help the search problem – who knows, maybe search is their business model – it just took a search guy at yahoo to point it out….