What Site Owners, Web Developers & SEOs Should Know About The Yahoo Microsoft Deal

By now, everyone has read all about the news that Yahoo is replacing its search index with Microsoft’s Bing. In a way, it’s a great story of complete reversal, as in 2002, Microsoft didn’t have its own index and instead used Inktomi. Late that year, Yahoo! acquired Inktomi, which spurred Microsoft to start building its own search index to avoid having a search supplier owned by a major competitor. Now Yahoo is ditching its index (including all of the technology it acquired with Inktomi) to use the very index it motivated Microsoft to build.

We’ve read about what this means for advertisers (more overall traffic from the combined audience, use of Microsoft adCenter for self-serve and Yahoo!’s sales force for premium) and for searchers (they likely won’t notice), but what does the deal means for those who create websites: publishers, web developers, and SEOs?

Web Developers

The hardest hit by this change will likely be developers. Over the last couple of years, Yahoo seems to have shifted its focus from innovating the search index to innovating its developer offerings: encouraging third-party development and creating a “developer ecosystem” for search.

Any developer options that don’t rely on the Yahoo search index may be unscathed. In particular, the non-search development tools and search-related offerings that are solely focused on the user interface may continue to be supported. While Bing will power Yahoo’s search engine, Yahoo will control their user interface and likely will try to continue to differentiate there. That’s means Search Monkey, which enables site owners to enhance how their results appear on Yahoo, is potentially safe.

Build Your Own Search Service (BOSS) likely won’t be so lucky. BOSS is built on the Yahoo index as its foundation. A company can build their own search engine using Yahoo’s underlying technology and differentiate via the user experience.  Essentially, that’s what Yahoo is planning to do now with Bing as their underlying technology. No more Yahoo index likely means no more BOSS. Yahoo all but concedes as much: “We can tell you that BOSS will remain live for the time being.” What does that mean for companies like hakia, OneRiot, Daylife, and Cluuz? And for that matter, all of the developers using BOSS who are now filling the Yahoo BOSS message boards with questions?

On the Yahoo developer blog, Yahoo commented that “For SearchMonkey and BOSS, we currently do not have anything concrete to tell you. Clearly, we’ll need to work with Microsoft to determine what makes the most sense for you and for us.” If BOSS’s future is left up to Microsoft, I have no doubt that future will involve migrating BOSS users to the Bing search API. In order to continue to support BOSS, Microsoft would have to completely recreate it to work with the Bing search infrastructure. Why would they do that when they can increase the audience of a product they already have? It’s possible they’ll add some of the unique BOSS features their search API (such as unlimited queries, ability to mash up the data with other sources, and ability to tweak ranking signals), but I wouldn’t hold my breath. The Yahoo BOSS team is just as in the dark as the developers wanting answers. From a message board post: “What specifically does it mean for BOSS? Honestly the team is still absorbing the implications and we just don’t know.”.

BOSS users could switch to Google’s Custom Search API, but it is more restrictive than Microsoft’s offering, and isn’t really well-suited as the foundation of a search engines or other commercial company. Several other companies offer web indices, such as CommonCrawl and Alexa, so perhaps they or a new company will take advantage Boss’s imminent demise and offer matching features.

Any Yahoo offerings that don’t rely on an underlying index, such as the Yahoo User Interface library are likely going to remain. Yahoo confirmed this in their blog post:

“We’ve also received questions about the future of Yahoo!’s other developer offerings, such as YUI, YQL , and Pipes. We wanted to let you know that today’s news does not affect these products. None of our other non-search developer products are affected.” [Emphasis mine.]

However, look for any search index-based offerings (such as the Maps API and Local API) to be deprecated in favor of the Bing equivalents once the deal goes through.

Search Engine Optimizers and Site Owners

What about those who are concerned with getting customer acquisitions through organic search? How will this change impact them? From a traffic perspective, take a look at how well you’re indexed and ranked in Bing. That’s how well you’ll be indexed and ranked in Yahoo. What do your titles and descriptions look like in the results in Bing? That’s how things will generally look in Yahoo. This might not be a bad thing for site owners, as over the last year, Yahoo’s search quality seems to have been declining to the point that I’ve been wondering if their engineering team had already begun to be phased out or least was spending a lot of time at the bar mourning the likely phase out.

Just as you don’t need to optimize separately for AOL since they use Google’s index, you won’t need to optimize for Yahoo since they’ll use Bing’s index. The exception to this may be in how Yahoo displays results. We’ll have to wait and see exactly what this means, but Bing has been trying to differentiate in display and it supposedly, Yahoo will continue to do that as well. This may mean, for instance:

• SearchMonkey will continue to be important as a way to stand out in the results.
• Hmm. I can’t really think of anything else.

My guess is that the robots-nocontent tag will no longer be supported, since Bing’s infrastructure doesn’t support it. The search engines have already come together to standardize their support of robots.txt and XML Sitemaps, so site owners shouldn’t worry about changing anything with those.

The bigger issue many SEOs are concerned about is Site Explorer. Site Explorer is one of the more reliable tools for competitive backlink research. You can see a substantial list of links to any site, generally ordered according to value. That’s useful stuff! Both Google and Bing Webmaster Tools provide backlink data, but only for your own sites. Yahoo will be unable to maintain Site Explorer without a search index of their own. Will Bing take it over? Well, it could add the feature to its Webmaster Tools, but Microsoft has historically been moving the other direction. They removed the ability to query their index for link data with the link: operator in 2007 and have never brought it back for competitive research.

Microsoft likely won’t be motivated to add a feature that they specifically chose to remove. And it’s not trivial to build the code to query for competitive links and store the data. Believe me, I know. I managed the process for adding non-competitive backlink data to Google Webmaster Tools. As with the potential end to BOSS, the potential end to Site Explorer opens up new opportunities for third-parties. In fact, the same companies who build a web index could provide competitive link data. Currently, SEOmoz  provides Linkscape, which offers some similar features. (Speaking of SEOmoz, Rand Fishkin posted yesterday about the SEO impact of this deal.) Majestic SEO and Exalead provides link data as well.

More generally, will Microsoft step up its efforts with webmaster relationships? Yahoo used to have a fairly significant presence in the community. In addition to Site Explorer, they were a constant at conferences and participated in online discussions. That participation has declined lately, coinciding with the decline in search quality. Microsoft seemed to be rallying with its webmaster relationships with the launch of the Webmaster Center in November 2007. But Microsoft hasn’t updated the Webmaster Center with new features since August 2008. (A minor release in November didn’t add  new features).

Microsoft didn’t respond to my questions about their current and future resource investment in this area. They did recently release a rudimentary SEO Toolkit, although it requires Windows Vista and IIS 7.0 to run.

And what about paid inclusion? Yahoo has long offered Search Submit Pro, which essentially enables sites to pay to be included in the organic listings. Microsoft doesn’t offer a similar product and while it’s certainly possible that Microsoft will add this product to their offerings, paid inclusion is quite a substantial shift in overall approach to organic search. It’s less about the ability to implement the technology and more about belief around what constitutes an “organic” index. Danny Sullivan asked about paid inclusion at the announcement press conference. Carol Bartz, Yahoo CEO replied, ” Paid inclusion, we’ll decide on that later.” But it would be difficult for Yahoo to continue the program on its own, as Yahoo will no longer have control over what pages are included in the search index.

In the end, it’s about the traffic

The big question is will this partnership significantly change market share percentages? Depending on whose numbers you use, Google has either 65% or 74% share in the US (more in some European countries). That puts the combined Yahoo/Microsoft share at 28% or 25.5%. That’s substantial traffic, sure, and worth paying attention to. But what will the share look like in three years once the deal is done and we barely remember Yahoo ever had its own index? My guess is pretty similar to how it looks now. Except Google will probably have slightly higher share. I just don’t see anything game changing here that will cause a mass exodus from the status quo. But I’ve been wrong before. What I do know is that site owners who have ignored how their sites were doing in Bing until now do to low traffic numbers will likely start paying a lot more attention.