By now, everyone has read all about the news that Yahoo is replacing its search index with Microsoft’s Bing. In a way, it’s a great story of complete reversal, as in 2002, Microsoft didn’t have its own index and instead used Inktomi. Late that year, Yahoo! acquired Inktomi, which spurred Microsoft to start building its own search index to avoid having a search supplier owned by a major competitor. Now Yahoo is ditching its index (including all of the technology it acquired with Inktomi) to use the very index it motivated Microsoft to build.

We’ve read about what this means for advertisers (more overall traffic from the combined audience, use of Microsoft adCenter for self-serve and Yahoo!’s sales force for premium) and for searchers (they likely won’t notice), but what does the deal means for those who create websites: publishers, web developers, and SEOs?

Web Developers

The hardest hit by this change will likely be developers. Over the last couple of years, Yahoo seems to have shifted its focus from innovating the search index to innovating its developer offerings: encouraging third-party development and creating a “developer ecosystem” for search.

Any developer options that don’t rely on the Yahoo search index may be unscathed. In particular, the non-search development tools and search-related offerings that are solely focused on the user interface may continue to be supported. While Bing will power Yahoo’s search engine, Yahoo will control their user interface and likely will try to continue to differentiate there. That’s means Search Monkey, which enables site owners to enhance how their results appear on Yahoo, is potentially safe.

Build Your Own Search Service (BOSS) likely won’t be so lucky. BOSS is built on the Yahoo index as its foundation. A company can build their own search engine using Yahoo’s underlying technology and differentiate via the user experience.  Essentially, that’s what Yahoo is planning to do now with Bing as their underlying technology. No more Yahoo index likely means no more BOSS. Yahoo all but concedes as much: “We can tell you that BOSS will remain live for the time being.” What does that mean for companies like hakia, OneRiot, Daylife, and Cluuz? And for that matter, all of the developers using BOSS who are now filling the Yahoo BOSS message boards with questions?

On the Yahoo developer blog, Yahoo commented that “For SearchMonkey and BOSS, we currently do not have anything concrete to tell you. Clearly, we’ll need to work with Microsoft to determine what makes the most sense for you and for us.” If BOSS’s future is left up to Microsoft, I have no doubt that future will involve migrating BOSS users to the Bing search API. In order to continue to support BOSS, Microsoft would have to completely recreate it to work with the Bing search infrastructure. Why would they do that when they can increase the audience of a product they already have? It’s possible they’ll add some of the unique BOSS features their search API (such as unlimited queries, ability to mash up the data with other sources, and ability to tweak ranking signals), but I wouldn’t hold my breath. The Yahoo BOSS team is just as in the dark as the developers wanting answers. From a message board post: “What specifically does it mean for BOSS? Honestly the team is still absorbing the implications and we just don’t know.”.

BOSS users could switch to Google’s Custom Search API, but it is more restrictive than Microsoft’s offering, and isn’t really well-suited as the foundation of a search engines or other commercial company. Several other companies offer web indices, such as CommonCrawl and Alexa, so perhaps they or a new company will take advantage Boss’s imminent demise and offer matching features.

Any Yahoo offerings that don’t rely on an underlying index, such as the Yahoo User Interface library are likely going to remain. Yahoo confirmed this in their blog post:

“We’ve also received questions about the future of Yahoo!’s other developer offerings, such as YUI, YQL , and Pipes. We wanted to let you know that today’s news does not affect these products. None of our other non-search developer products are affected.” [Emphasis mine.]

However, look for any search index-based offerings (such as the Maps API and Local API) to be deprecated in favor of the Bing equivalents once the deal goes through.

Search Engine Optimizers and Site Owners

What about those who are concerned with getting customer acquisitions through organic search? How will this change impact them? From a traffic perspective, take a look at how well you’re indexed and ranked in Bing. That’s how well you’ll be indexed and ranked in Yahoo. What do your titles and descriptions look like in the results in Bing? That’s how things will generally look in Yahoo. This might not be a bad thing for site owners, as over the last year, Yahoo’s search quality seems to have been declining to the point that I’ve been wondering if their engineering team had already begun to be phased out or least was spending a lot of time at the bar mourning the likely phase out.

Just as you don’t need to optimize separately for AOL since they use Google’s index, you won’t need to optimize for Yahoo since they’ll use Bing’s index. The exception to this may be in how Yahoo displays results. We’ll have to wait and see exactly what this means, but Bing has been trying to differentiate in display and it supposedly, Yahoo will continue to do that as well. This may mean, for instance:

• SearchMonkey will continue to be important as a way to stand out in the results.
• Hmm. I can’t really think of anything else.

My guess is that the robots-nocontent tag will no longer be supported, since Bing’s infrastructure doesn’t support it. The search engines have already come together to standardize their support of robots.txt and XML Sitemaps, so site owners shouldn’t worry about changing anything with those.

The bigger issue many SEOs are concerned about is Site Explorer. Site Explorer is one of the more reliable tools for competitive backlink research. You can see a substantial list of links to any site, generally ordered according to value. That’s useful stuff! Both Google and Bing Webmaster Tools provide backlink data, but only for your own sites. Yahoo will be unable to maintain Site Explorer without a search index of their own. Will Bing take it over? Well, it could add the feature to its Webmaster Tools, but Microsoft has historically been moving the other direction. They removed the ability to query their index for link data with the link: operator in 2007 and have never brought it back for competitive research.

Microsoft likely won’t be motivated to add a feature that they specifically chose to remove. And it’s not trivial to build the code to query for competitive links and store the data. Believe me, I know. I managed the process for adding non-competitive backlink data to Google Webmaster Tools. As with the potential end to BOSS, the potential end to Site Explorer opens up new opportunities for third-parties. In fact, the same companies who build a web index could provide competitive link data. Currently, SEOmoz  provides Linkscape, which offers some similar features. (Speaking of SEOmoz, Rand Fishkin posted yesterday about the SEO impact of this deal.) Majestic SEO and Exalead provides link data as well.

More generally, will Microsoft step up its efforts with webmaster relationships? Yahoo used to have a fairly significant presence in the community. In addition to Site Explorer, they were a constant at conferences and participated in online discussions. That participation has declined lately, coinciding with the decline in search quality. Microsoft seemed to be rallying with its webmaster relationships with the launch of the Webmaster Center in November 2007. But Microsoft hasn’t updated the Webmaster Center with new features since August 2008. (A minor release in November didn’t add  new features).

Microsoft didn’t respond to my questions about their current and future resource investment in this area. They did recently release a rudimentary SEO Toolkit, although it requires Windows Vista and IIS 7.0 to run.

And what about paid inclusion? Yahoo has long offered Search Submit Pro, which essentially enables sites to pay to be included in the organic listings. Microsoft doesn’t offer a similar product and while it’s certainly possible that Microsoft will add this product to their offerings, paid inclusion is quite a substantial shift in overall approach to organic search. It’s less about the ability to implement the technology and more about belief around what constitutes an “organic” index. Danny Sullivan asked about paid inclusion at the announcement press conference. Carol Bartz, Yahoo CEO replied, ” Paid inclusion, we’ll decide on that later.” But it would be difficult for Yahoo to continue the program on its own, as Yahoo will no longer have control over what pages are included in the search index.

In the end, it’s about the traffic

The big question is will this partnership significantly change market share percentages? Depending on whose numbers you use, Google has either 65% or 74% share in the US (more in some European countries). That puts the combined Yahoo/Microsoft share at 28% or 25.5%. That’s substantial traffic, sure, and worth paying attention to. But what will the share look like in three years once the deal is done and we barely remember Yahoo ever had its own index? My guess is pretty similar to how it looks now. Except Google will probably have slightly higher share. I just don’t see anything game changing here that will cause a mass exodus from the status quo. But I’ve been wrong before. What I do know is that site owners who have ignored how their sites were doing in Bing until now do to low traffic numbers will likely start paying a lot more attention.

Yesterday Yahoo spoke about taking search much more into the real world by turning “web pages” into “web objects.” Today there were a couple of announcements at the Where 2.0 conference related to identifying unstructured data online and linking it according to places and locations — in other words made geo-relevant.

Tyler Bell of Yahoo gave the talk at Where about a “geo-enriched location-aware internet.” Where is a very developer-focused event. Bell’s talk was essentially about building the location “infrastructure” that will help support the vision articulated at the Yahoo Search event yesterday. The first aspect of the talk was about  Yahoo GeoPlanet (which provides layers of geographic identifiers and place-based information to developers). Bell announced that in addition to offering an API, GeoPlanet will now allow developers to download all the data for free (with attribution to Yahoo) and host the data themselves.

Yahoo also announced Placemaker. Placemaker is an API that enables developers to pass unstructured data in documents to the service. It extracts the location information in those documents and allows the information to be associated with one or more locations (publishers/developers need to decide which places are most important). MetaCarta offers a similar capability.

Absent my attendance at the Yahoo Search event yesterday these would have seemed fairly map-geeky announcements. But in the context of Yahoo’s larger project to more directly tie search and the internet to the physical world this was quite interesting.

Here’s more information from the Yahoo Geo Blog.

Yahoo has added a few new tools to its BOSS platform that the company says will give developers more insight to their search results and help build unique search experiences.

The new tools include:

• Integration with Delicious content – end-users can get Delicious saves, top tags, and count information within their search results
• Advanced language capabilities – developers can now filter specific language results for a given market. BOSS now supports Czech, Hungarian, and traditional Chinese
• New sorting functionality – BOSS News service can now sort by date and/or a specified time range in days, weeks, or hours

Yahoo’s announcement has more information about all three tools, including a few code examples.

Today, Yahoo! has announced several new features to their Build Your Own Search Service (BOSS) platform as well as usage fees. When BOSS launched in July 2008, the stated goal was to spur innovation and disrupt the search market. Will the additional of usage fees hinder that goal and become a barrier to entry for the cost-limited startups Yahoo! was originally trying to help? Yahoo says no, that they “believe that introducing the proposed pricing structure will improve the ecosystem by optimizing capacity for our serious developers.”

Fee structure

The BOSS charges kick in only once queries exceed 10,000 a day. After that, there’s a tiered model based on what is requested through the API. The pricing will go into effect next quarter.

New features

In exchange for the introduction of fees, BOSS has added several new features. Key among them is the ability to monetize results. Yahoo! has adjusted the terms of service to allow developers to use third-party monetization platforms in conjunction with BOSS results. This is a shift from Yahoo!’s initial plan, which was to provide Yahoo-powered ads with a revenue sharing model. When BOSS first launched, Yahoo told me that they were planning to require Yahoo ads at some point and that “over the next several months, a BOSS monetization capability, using Yahoo! search advertising and potentially other models, will be made available for partners and developers to create a search revenue stream for their business.”

In addition, Yahoo! is expanding the maximum number of results returned from 50 to 1,000 and will be introducing an SLA to ensure reliability.

BOSS will be offering access to SearchMonkey data as well, starting with structured markup that the Yahoo! web crawler has extracted. This is yet another move to encourage the use of semantic standards on the web. (In a session at SMX West yesterday, Arnab Bhattacharjee, Senior Director, Yahoo! Search Technology and Engineering at Yahoo! reiterated Yahoo!’s encouragement of the use of structured markup.)

They’re also making longer abstracts of documents available (up to 300 words from the previous 170).

New functionality for webmasters: link data

The most interesting  new feature for webmasters may be the ability to query Site Explorer link data through the BOSS API. Presumably, however, the number of results returned will be limited to 1,000,  which may not completely satisy those looking for backlink data.

While the BOSS improvements are encouraging, it remains to be seen if the new pricing structure slows down Yahoo!’s efforts to disrupt the market by jump starting those with little cash.

Yesterday, we got a new news search engine which taps into Twitter that Yahoo’s excited about, as it uses their BOSS system. That kind of annoyed me. Why didn’t Yahoo just build the service themselves? Or a regular Twitter search service, for that matter? And where’s Twitter search from Google?

BOSS is Yahoo’s Build Your Own Search Service product, which lets anyone take Yahoo’s data and create their own search engine. It’s grown into part of Yahoo’s “open strategy” where it hopes to make gains on the web by supposedly pulling in data from everywhere as well as being open to others using its data.

TweetNews was trotted out by Yahoo as the latest example of how BOSS is succeeding. It takes stories showing up on Yahoo News and ranks them based on how much Twitter activity they’re receiving.

Kind of cool, right? Sure, it is. But then again, TweetNews was created by Vik Singh, a Yahoo programmer who works on the BOSS team. Why not just build this as Yahoo product itself? Hosted on a Yahoo site, rather than depending on Google’s App Engine (and dying when it exceeded the allowed quota). Moreover, why not build Yahoo’s own version of Twitter search?

People can argue over whether Twitter has mass appeal or not. You cannot argue that at this point, news DOES break out on Twitter far ahead of any other source. I watched firsthand how news about fires breaking out around me in Southern California last November was reported on Twitter before it hit Google News and other sources. The same is true for two different earthquakes I’ve gone through in the past six months. There are any number of other examples where Twitter has news that’s coming out and being eventually cited. Yesterday’s New York plane crash on the Hudson River was yet another case in point.

Twitter has powerful, compelling material. And if you want to find that material, you go to Twitter. Not to Yahoo. Not to Google, the king of search. Twitter.

I think it’s a missed opportunity for both Google and Yahoo. And for Yahoo in particular, while Google is closing down some search projects (arguably for good reasons), Yahoo should still be building its own unique and valuable search tools — not depending so much on others to do it.

The Yahoo Search Blog announced that BOSS, Yahoo’s build your own search service, is now serving 10 million queries per day. That is about 100 queries per second and according to Yahoo, who cites comScore, that is just behind Ask.com’s search volume.

Note, these queries do not count as part of Yahoo’s search network, because they are technically served at third party web sites.

We asked Yahoo which services were using BOSS most often, which queries were most popular and so on. Yahoo was unable to give us an answer, due to not having their partner’s consent. It would be nice to see how searchers are utilizing this service, i.e. top search sites, top search queries and so on.

In July Yahoo! launched their BOSS API, which enables you to “build your own search service”. Today, they’ve expanded the functionality of that API by adding a Key Terms feature. The technology used in Key Terms is the same used for Search Assist, which provides search suggestions and enables searchers to explore concepts related to the query.

Yahoo! says they’re adding this feature in response to developers’ requests for greater access to “deep infrastructure assets”. The Key Terms feature uses term frequency and positional and contextual heuristics to return ordered lists that describe a web page. Each result returned for a query includes associated metadata of up to 20 terms that describe that result.

In the blog post announcing the feature, Yahoo! uses a sample query of [obama] to illustrate that the first result, change.gov, includes key terms metadata such as Vice President-elect, President-elect, American, and lobbyists.

Yahoo!’s post explains that the feature can be applied in other ways than illustrated by Search Assist and that it can be used for semantic analysis, relevancy models, and to categorize similar documents. They note that Yahoo! developer Ted Drake used Key Terms to power the “related terms” feature of his food-related search engine tartin3.com.

Since the launch, Yahoo! has been showcasing uses for BOSS technology. Generally, these have been different visual representations of results and filtered results.

Last week, Microsoft Live Search launched a fuller-featured API, which also opens up access to their data. Search engines have an amazing amount of data and intelligence about the web and it’s a great trend that they’re beginning to make that available to developers as the foundation for innovative web applications.

Yahoo! has just unveiled the next phase in their plan to spur search innovation by providing search-related resources to developers. The Yahoo! Build Your Own Search Service (BOSS) enables developers to access Yahoo! search results, combine them with other sources, rerank them, and define their appearance. Yahoo! says they are making BOSS available in an attempt to spur innovation in the search space and disrupt the market. They point out that unlike with other web companies, a search startup has many obstacles: from cost (it takes a lot of machines to process all the web’s data) to expertise (some of the world’s smartest PhDs work at the major search engines) to historical data (which search companies can only get a hold of by that old-fashioned method of waiting).

Below, more on what Yahoo! is making available through BOSS and how likely it is that the launch will really create the next Google.

Is BOSS different from the search APIs offered by Google and Microsoft? 
On first glance, the BOSS API seems somewhat similar to Google’s custom search API and Microsoft’s Live Search API. It enables developers to request search results (from web search, news, and images), reorder them, and style them. However, Yahoo! points out key differences, mostly based on the overall intent of the BOSS program (to power new search startups). Yahoo’s API allows unlimited queries, a necessary feature for developers who use the API to build a search engine. And it allows for mashups of its data with other data sources. Hakia, for instance, is using the API to blend Yahoo! results with their own and runs all results through their proprietary algorithm, SemanticRank. And they are displaying Yahoo!’s image results in a mashup with their own web index results. Neither Google nor Microsoft allow such flexibility, and both require branding of the search results.

Yahoo! doesn’t provide access to its ranking signals, but it does allow developers to add their own signals to the set of results. Me.dium is using the API to order results based on social signals, notably what pages its users have accessed recently.
Yahoo! provided two examples of custom ranking: search results reordered for popularity based on matches to popular Delicious results and search results reordered based on topical matches to recently edited Wikipedia pages. Below, you can see the before and after results for the recency reordering.

On the one hand, intent isn’t as important as practical application, and the current feasible applications of BOSS seem somewhat similar to what Google and Microsoft offer. Its current feature set seems ideal for a comprehensive site search implementation, for example. However, intent does become important when considering how Yahoo! might evolve the program, and they say this is only the first phase of planned features. They say they’re looking to the developer community to determine what their roadmap should be. Yahoo! is explicitly looking to disrupt the search market by helping search startups overcome some of those obstacles inherent in the search business. They reduce the burden of crawling and the limitations caused by a lack of historical data by providing search results, and they allow for innovation by enabling developers to create their own ranking, look and feel, and mashups.

Will BOSS power innovation in search? 
Those who are truly looking to innovate in the search space may feel that they need access to the raw content of a web index, not simply to the results.

Hakia, who is a satisfied customer of the API, isn’t replacing their crawl. They need to crawl the web themselves to implement their natural search innovations using what they term “QDEXing”. Hakia president Melek Pulatkonak told me that “Hakia views the BOSS initiative as a means of accelerating our efforts to QDEX the entire Web, and therefore become the first full-scale semantic search engine. Yahoo! Search BOSS is the best partnership offer for developments like the one in Hakia, and is an unprecedented initiative in the market." (It remains to be seen if Microsoft’s acquisition of Powerset will speed up or slow down competitor Powerset’s roadmap.)

Me.dium is also primarily using its own index, and in any case, isn’t looking to topple Google. As Chris Sherman points out, they’re more akin to StumbleUpon. However, both implementations highlight ways the API could help developers try innovative things in search, particularly if those innovations revolve around ranking and display.

What’s the revenue model?
In addition to search startups needing more access than BOSS provides, they also may want to control their revenue stream. While BOSS isn’t launching with ads, and developers can monetize any way they want for now, Yahoo! does plan to require Yahoo! ads be displayed beside the search results at some point. Yahoo! told me that “Over the next several months, a BOSS monetization capability, using Yahoo! search advertising and potentially other models, will be made available for partners and developers to create a search revenue stream for their business.” At that point, developers will be locked into using whatever Yahoo! provides.

Are there other options?
Yahoo! isn’t the only company looking to spur innovation by making large-scale data available. For instance, CommonCrawl says that their mission is to "build, maintain and make widely available a comprehensive crawl of the Internet for the purpose of enabling a new wave of innovation, education and research" and plan to operate as a non-profit. This project isn’t aimed at powering search innovation specifically, but rather at the entire realm of information fields. Gil Elbaz, founder of CommonCrawl, told me, "we think a common crawl of the web will be a great resource for anyone trying to innovate in information-related research."

Alexa also makes a web index available, although it’s much smaller than Yahoo!’s and isn’t free.

Yahoo! is providing a separate API for academic use. For now, the data available is the same as the public API but will provide more results per API request (1000 results rather than 50). A “custom” API is still in the works for which Yahoo! will work more closely with partners.

Uses other than consumer search?
I asked Yahoo! how they felt about other uses of this API. For instance, search marketers can think of many tools that could be powered by search results data. For now, the terms of service requires the data be used only for consumer web search.

“You are permitted to use the Services only for the purpose of incorporating and displaying Web Search Results from such Services as part of a Search Product deployed on your Web site (“Your Offering”). A “Search Product” means a service which provides a response to a search query, keyword or other request served from an index or indexes of data related to Web pages generated, in whole or in part, by the application of an algorithmic search engine.“

They’ll be monitoring use by looking at things such as how many queries resulted in clicks. However, they said that they were looking at potentially making other data offerings available that may be of interest to SEOs.

Overall, this is an interesting idea from Yahoo! Can it shake up the status quo market share? I’m not so sure about that. But it is another sign of Yahoo!’s commitment to the developer community and of their willingness to think creatively about market share (although they may be thinking more about ways to find distribution channels beyond toolbar deals than they are about helping competing search engines be successful).

Yahoo’s relatively new CTO Ari Balogh expanded upon Yahoo’s “open” initiatives in a keynote speech to the Web2.0 conference yesterday. There’s a very extensive discussion, including a video of the keynote, on the TechCrunch site (additional coverage in Search Engine Journal). The SearchMonkey beta is the first step in executing on the strategy, which will roll out in increments over the next several months.

The “OS” part is not “operating system” but “open strategy.” Beyond openness, the themes of Balogh’s talk were making Yahoo more social, more consistent and unified, more personal, and even more “portable.”

Rather than creating another social network, Yahoo will be seeking to create a single profile that helps ground and center the social experience across the network. This is what has been missing at Yahoo — a social center (360 failed to gel and provide that). YOS also seeks to make the whole of Yahoo more social by overlaying relevant contacts and discussions on different pages within Yahoo, including the home page. To that end, there’s an emphasis on what amount of widgets (though that word isn’t used) can be placed throughout Yahoo on various pages. The whole strategy extends into mobile (consistent with the spirit of Yahoo onePlace).

Balogh also sought to court developers with tools that make it easy to develop on top of Yahoo, the platform. Balogh used the phrase “rewiring Yahoo” several times, indicating a range of technical changes on the back end to make APIs consistent and make the development environment more coherent and unified across the network.

In fact, the vision is impressively coherent and, if it can be pulled off, offers a successful differentiation of Yahoo from its competitors. It moves Yahoo in the direction of Facebook (a social platform for third-party developers) but with all the Yahoo traffic and content assets.

In one sense, it’s the consumer-facing version of the big vision that surrounds the AMP initiative on the advertiser side.

The looming question, however, is whether Yahoo will get to see these aspirations fulfilled if Microsoft buys the company, especially in light of ambitious, somewhat parallel initiatives by the latter.

The Yahoo Search Blog announced a new WordPress plugin that automatically creates Yahoo Shortcuts for the words you write in a blog post.

If you mention a stock ticker, that word on your blog will typically contain a mouse-over effect that will display a Yahoo Finance chart. Or if you have an address, they might show you a Yahoo Map. Yahoo integrated the plugin with web search, maps, finance, product search, Flickr, Yahoo Autos, news search, and more. For more details, visit http://shortcuts.yahoo.com/.

Oh, and if you are one of the first 500 bloggers to integrate this into your WordPress blog, you can get a free t-shirt.