Google Realtime Search & The Aftermath Of The Google-Twitter Split

Last Friday, Twitter quietly shutdown its “firehose” of tweet data that was being piped to Google. Like a gas station no longer getting deliveries, Google in turn effectively had to hang a “Closed” sign on its Google Realtime Search service. What happened, and what’s next for those who depended on Google to get some of their Twitter gas? Some thoughts and advice, below.

Topsy Provides Twitter Archive Search

First, there’s a great alternative to Google Realtime Search: Topsy.

Indeed, the company has just put out a blog post reminding the world that it’s the only Twitter archive search service left standing.

Fair enough. That’s totally correct. The company expanded its coverage back August 2010, and my review below explains more about some of the features it offers:

• Topsy: Now Searching Tweets Back To May 2008

In fact, Topsy allowed you to go farther back than Google did. Google had promised that it would extend its index earlier than February 2010, but I don’t think that really happened.

Topsy tells me its index still goes back to May 2008, as I originally reported.

Bing Has The Firehose, But No Real Archive

Unlike Google, Topsy still has access to that “firehose” data from Twitter (and won’t reveal any more details than that). That’s why it’s still ticking along.

Microsoft’s Bing search engine also has firehose access. However, Bing Social Search doesn’t really let you go back more than a few days.

Bing’s tool is more designed, like Twitter’s own search engine, to allow you to search about what’s currently being said through Twitter and other update services at the moment. It’s not aimed at providing some type of historical search service.

Tweet Origin Tools

Some of those missing Google Realtime Search may be trying to track a popular topic on Twitter back to its origin. What The Trend may help here, and I’ll try to gather some others in the future. Here are also some past articles on this topic:

• The Origin Of #nickcleggsfault & Getting To More Relevant Tweets
• Who Started The “Dear Yahoo, I’ve Never Heard Anyone Say ‘Let’s Yahoo It’ Sincerely Google” Twitter Joke?

Where’s Twitter’s Own Archive?

At this point, you may be wondering why Twitter doesn’t make it possible to search through its own tweets for as far back as it has them. Yes, that does seem kind of crazy. However, it’s a conscious decision that Twitter has made.

Twitter has repeatedly told me, and others, that it wants to create search products that it thinks are more important to its users and that partners aren’t providing.

As Mike Abbott, Twitter’s vice president of engineering told me last year:

Google doing it [archive search] takes some of the pressure off. Where do we want to innovate in this world and drive unique set of experiences?

There’s no doubt that Twitter has build some great search tools. These articles have a bit more about that:

• The New Twitter & Search, An Illustrated Guide
• Move Over Time Sorting: Twitter Gets “Top Tweets” Search Results

And these articles talk more about the issue with archive searching and Twitter in general:

• Where Have All The Old Tweets Gone?
• All The Old Tweets Are Found: Google Launches Twitter Archive Search

As For The Library Of Congress…

By the way, you may recall that Twitter has been sending tweets to the US Library Of Congress. While that is an archive of sorts, it’s not one that anyone can search.

Also, just a little privacy reminder. While you can delete tweets, you’ve effectively only got six months from when you make a public tweet to prevent it from being stored with the Library Of Congress.

There’s a six month delay in the data they receive. After that, there’s no mechanism to prevent your tweets from later being discovered by Logan and Jessica when they stumble into the ruins of DC in the distant future.

What’s Left At Google?

Google still has access to any tweets that it finds through its regular crawling of the web. That means if you’re doing a regular Google search, you might find tweets that way. It just won’t be as focused, so you might find it helpful to use some of the search techniques covered here:

• Where Have All The Old Tweets Gone?

I’m checking to see if Google has any hidden commands that might help. One of the best is doing:

site:twitter.com/accountname

That type of search restricts a search to tweets from a particular person.

Expect Delayed Tweets At Google

However, in trying that today, you can already see problems that Google’s having now with tweets:

These are all tweets that I made yesterday. Nothing I tweeted this morning (I’ve done at least four tweets) is showing up. Worse, you can’t even tell what these tweets are about, as there’s been no title automatically created for them.

When I went looking for one particular fresh tweet of mine, I couldn’t find it, though oddly, I did get shown someone retweeting it:

Annoyingly, I’ve also found some cases where aggregators show up when my own tweet doesn’t:

That leads over to the Inagist site, which I never heard of before, and which apparently embeds the photo I uploaded through Twitter to yfrog. Or something. It kind of makes my head hurt.

All I know is that I don’t find my tweet, which is a problem for Google, but also for Twitter. But let’s stick with the Google problems, for now.

Twitter, Google & News Shares

Google also uses Twitter data in a variety of other ways. One way had been to show the number of shares of news articles or updates that people were doing related to a news topic. Some examples of these are in our article from last October:

• In The Wake Of Bing & Facebook, Google Web Search Tests Getting More Social

Looking today, I see less of this. But occasionally, these do appear:

If you try to drill in, you get an error:

Twitter, Google & Social Search

Google also taps into Twitter for its Google Social Search service, both to help create connections and to help surface content that is being shared on Twitter by those in your network. Our story from February covers this more with some examples:

• Google’s Search Results Get More Social; Twitter As The New Facebook “Like”

Looking today, I can still see this working, where Google is clearly seeing things that are shared via Twitter through its ordinary crawling.

Look at the last line below, and you can see how Google flags this story as being shared on Twitter:

But interestingly, I also noticed something new today:

There, you can see how when I hover over “Matt Cutts” in the “shared this” area, I’m told I’m connected to him through the new Google+ social network.

While Google+ has mainly seemed a way for Google to collect data it feared being locked up within the walls of Facebook (see Steven Levy’s excellent Wired piece for Google effectively confirming this), it suddenly is providing a useful backup for Twitter, too.

For now, that backup mainly seems to be helping in forming social connections. But in the future, it could be that Google Realtime Search might return powered by posts from Google Plus.

Loss Of Link Juice

Finally, Google has used the sharing on Twitter as a form of ranking signal to help determine the quality of content it lists. This was a bigger impact for results in Google Realtime Search, but it was also used in other ways. Our story below has more on this:

• What Social Signals Do Google & Bing Really Count?

Over at SEOmoz, they’ve been trying to test what if the loss of the firehose may have impacted SEO efforts. I think the results are fairly inconclusive, but you may want to check them out.

One big change is that tweeted links are back to being nofollow — IE, not passing link credit.

As my What Social Signals Do Google & Bing Really Count? article explains, in the Twitter firehose, links didn’t have nofollow attached. That’s a lot of link juice that’s just evaporated. It’s unclear what the impact will be for publishers and Google alike, yet.

Says Google…

I talked with Google’s Amit Singhal — who oversees all of Google’s search products — about the impact on the Twitter firehose being closed. He said that Google won’t catch tweets as quickly as in the past, though he said the delay would be of one of shifting from seconds to minutes. My testing above suggests it’ll be much longer than that, in some cases.

Singhal also said that Google probably won’t have as comprehensive collection of tweets as it did in the past. While technically, Google has the capacity to crawl Twitter’s site and gather up all the tweets when they happen, he figured that would probably crash Twitter. Search engines generally try to be “polite” when crawling and not gather data so quickly as to impact a site’s human users.

In terms of social search, Singhal wasn’t certain the impact that the Twitter change might have on things yet. But he did say that Google was already having to calculate the number of shares, or tweets, that a particular page on the web may have gained on its own. That means Google can continue to create those counts, though it may take longer for it to understand the full counts and how quickly something is being shared.

I also asked about the loss of Google Realtime Search. It launched as part of a big Google press event, with some realtime results injected directly into the main results. There was a lot of fanfare over how important and useful this was. With it gone, isn’t Google losing something?

“Ideally, we would still have a partnership,” Singhal said. “But we’ve decided in all, we’re OK with the current state of things.”

Singhal also clarified that the firehose was turned off on Twitter’s side, as a result of the agreement not being renewed. Google felt that Google Realtime Search had to close entirely, because it depended so heavily on Twitter content, even though other realtime content was also part of it.

Do People Really Miss Google Realtime Search?

Google might be right about the “getting along OK” part. Yes, I miss Google Realtime Seach. Google tells me they’ve also had journalists begging for it to return and had to explain they can’t do anything without an agreement. Nicholas Carr penned a piece about having the “shakes” from this sudden realtime search withdrawal.

But in general, practically no one seems to be complaining. There was no barrage of “what’s up” tweets that came out when the service suddenly closed. In contrast, Google’s change of its navigation bar to black seems to have generated much more discussion.

My news editor Barry Schwartz, who constantly scans Google’s own forums and search forums across the web for his own Search Engine Roundtable site, tells me that he figures complains about realtime search being gone are only about 5% of those about the new navigation bar — if that.

It reminds me of when Google couldn’t reach a deal with the Associated Press last year. For about a month, AP content disappeared from Google News. Virtually no one noticed.

Still, a bigger test will come with breaking news events, I’d say. When actress Brittany Murphy died, it was the first big test of how realtime results could improve Google’s relevancy, and they very much did. Google seems to have lost something useful, I’d say.

It’s Also Twitter’s Problem

Of course, Twitter’s missing something, too. As I explained above, it’s not particularly nice to search for your own tweet and not be able to find it on Google, if that’s where you choose to look. Plenty will be looking there, I’d say, because Twitter has effectively trained them to do that. Nor has Twitter, so far, tweeted or posted anything about what people should do now if they want historical tweets.

More important, Google remains a powerful traffic driver. Now, instead of people finding tweets and ending up back at Twitter, they may show up at official aggregators or unofficial scrapers. That doesn’t seem to help Twitter’s bottom line, nor does it seem a good user experience.

Twitter, by the way, isn’t saying more than what I initially reported —

Since October 2009, Twitter has provided Google with the stream of public tweets for incorporation into their real-time search product and other uses. That agreement has now expired. We continue to provide this type of access to Microsoft, Yahoo!, NTT Docomo, Yahoo! Japan and dozens of other smaller developers. And, we work with Google in many other ways.

What Happened?

No one’s saying why the agreement was allowed to expire. While it was signed at the same time that Microsoft’s was, the Microsoft agreement is continuing. I get the impression this wasn’t because it was renewed but rather that the Microsoft deal hasn’t expired yet.

Microsoft told me, about about the deal:

We won’t disclose the terms of the deal, but it’s a long term arrangement that we’re pleased with, and plan to keep in place as long as it’s delivering benefit for people who use Bing.

I got a tip after the news broke that perhaps sheds more light. I was told that the rumor is, according to several CEOs who run search start-ups, that Google was negotiating to renew the agreement for two years at $35 million per year, or $70 million in total.

Now that’s not much for Google. But it’s likely a huge amount for Google given that the company pretty much doesn’t pay to license anything.

The last deal with Twitter, rumored to be for $15 million, was pretty unprecedented. Carrying Twitter’s ads on Google, with Twitter’s branding, definitely was. Twitter Promoted Tweets Come To Google explains more about this.

So maybe it wasn’t about the money as much as other issues. That leads to something else the tipster told me: that Microsoft is apparently not that happy with its Twitter relationship, not seeing the value in paying for it when smaller search startups get firehose access for free, and that it might just drop Twitter and license the data out from a third party.

Caveat time. I’ve not had a tip from this person before, so I can’t vouch for it with a history of this person always being right. It could, for all I know, be entirely off the mark. If anyone knows better and wants to share more, get in contact. I’d love to hear.

I did go back to Google, Microsoft and Twitter with this information. Twitter just reiterated what it said before. Google did the same. Microsoft gave me the statement above about its deal generally and said it doesn’t comment on rumor and speculation.

Actually, Microsoft does comment on rumor and speculation, as do Google and Twitter, whenever they decide its in their interests to do so.

I also asked Microsoft if there were any plans to create a comprehensive Twitter archive search and was told:

We’re not discussing future product plans, but we don’t have any immediate plans to create a deeper archive.

That’s all I know, at this point. If you’re looking for those older tweets, definitely check out Topsy. As for Twitter and Google, I guess it’s stay tuned.

Links Mentioned In The Article

• As Deal With Twitter Expires, Google Realtime Search Goes Offline
• Google Launches Real Time Search Results
• Brittany Murphy’s Death & Google’s Real Time Search Results
• Twitter Promoted Tweets Come To Google
• Where Have All The Old Tweets Gone?
• All The Old Tweets Are Found: Google Launches Twitter Archive Search
• Topsy: Now Searching Tweets Back To May 2008
• Bing Launches Social Search Features
• The New Twitter & Search, An Illustrated Guide
• Move Over Time Sorting: Twitter Gets “Top Tweets” Search Results
• An Illustrated Guide To Searching For Shared, Tweeted & “Realtime” Images
• The Origin Of #nickcleggsfault & Getting To More Relevant Tweets
• Who Started The “Dear Yahoo, I’ve Never Heard Anyone Say ‘Let’s Yahoo It’ Sincerely Google” Twitter Joke?
• Where’s AP In Google News? Apparently In Limbo, As Contract Running Out
• Google & AP Extend Long-Term Content Deal
• Google’s Search Results Get More Social; Twitter As The New Facebook “Like”
• Google’s Facebook Competitor, The Google+ Social Network, Finally Arrives
• What Social Signals Do Google & Bing Really Count?
• The Periodic Table Of SEO Ranking Factors