Yahoo announced that later this week the organic transition will commence for PC and mobile results in North America:

This is an important step toward our goal of improving the overall relevance of Yahoo! organic search results and attracting a larger audience to Yahoo! Search, to ultimately put your ads in front of more potential customers. You’ll want to make sure that you’re prepared for this change, so be sure to check out these tips and stay tuned to the Yahoo! Search blog for confirmation of when the organic search transition is complete.

A “powered by Bing” message at the bottom of results will indicate that these results are live.

Yahoo also said that limited testing of paid account transitions has begun. There will be a “transition” portal with “simple step-by-step [instructions for] creating a Microsoft Advertising adCenter account and importing your campaigns, or linking an existing adCenter account you may already have.”

Search Monkey will continue on but with substantial changes:

All of the existing enhanced result templates will continue to be generated from websites’ page markup and structured data feeds, and Yahoo! will continue to show this structured data on the Yahoo! Search results page, along with Microsoft’s organic listings. Over time, some of this structured data processing will be supported natively by the Microsoft platform. Webmasters will continue to have the ability to affect the presentation of a search result through page markup on their site (microformats and RDFa).

As we look to the future of Yahoo! Search, we are focusing on new search-related offerings we believe will provide additional value for publishers and partners.  In order to align our resources on strategic priorities, we have decided to close the SearchMonkey developer tool, gallery, and app preferences on October 1, 2010.  As a result, third party custom result apps, infobar apps, and data services will no longer appear on Yahoo!’s search results.  For developers who wish to retain their code, please export it using your favorite copy/paste tool before then.

Site Explorer will continue to be used by Yahoo until 2012 when the organic transition is complete:

[Y]ou should continue to provide your site information to Yahoo! using Site Explorer, because in many parts of the world search results will continue to be powered by Yahoo!’s systems until the full transition is complete for all markets by early 2012. This will help to ensure that you continue to get high quality traffic from searches originating on Yahoo! and our partner sites, even in markets that are not yet transitioned to Microsoft’s systems. We will share site information that you provide on Site Explorer with Microsoft during this transition period, to ensure that you get high quality traffic from search results that are powered by Bing also.

However Yahoo also recommends becoming familiar with the Bing Webmaster Center tool in the interim.

Related posts:

• All New Microsoft Bing Webmaster Tools
• What Site Owners, Web Developers & SEOs Should Know About The Yahoo Microsoft Deal

Postscript by Barry Schwartz: I spoke with Yahoo’s Shashi Seth after this announcement and I wanted to clarify that Site Explorer will ultimately be fully integrated into Bing Webmaster Center. Currently, the new Bing webmaster tools is missing a link tool completely. I suspect that was planned and Yahoo’s Site Explorer features will be that link tool in Bing Webmaster Center. We should know more about when Site Explorer’s integration schedule within the next 30 days or so.

Attempting to generate enthusiasm, he pressed on without slides: despite the popular misconception, Yahoo is not out of search, echoing statements made by CEO Carol Bartz and others on past company earnings calls. He would not have come to Yahoo six weeks ago if the company wasn’t committed to search and there wasn’t a great team there, and so on.

The power came back on and Seth was able to show his slides. There’s lots of innovation on the front end and with the user experience still to be done. The purple boxes below indicate Yahoo’s “contribution” to search after the Microsoft deal closes.

Seth then spoke at some length about Yahoo’s leadership in Asia and its mobile OEM and operator relationships in mobile around the globe.

Prabhakar Raghavan, SVP, Yahoo Labs/Search Strategy came next to again express the concept of the “web of things” vs. the “old” idea of the “web of objects” (documents). People don’t care about the index size anymore; they care about the right information or “answer.” He made the remark that 99 percent of search queries on Yahoo “have a noun” in them. This indicates that people are looking for information about the real world. The era of 10 blue links is dead, and so on.

Raghavan went on to detail Yahoo’s formidable academic and research successes: the top three texts about search are all by Yahoo scientists; the top prizes (papers) at all the 2009 and now 2010 (so far) data mining conferences have been won by Yahoo.

Larry Cornett, VP Consumer Products took the stage and reviewed many of the search innovations and rollouts of the past six months: Search Assist, Search Pad, breaking news/Twitter, richer Shortcuts, contextually relevant filters and refinements, multimedia search, etc. Cornett also previewed some future innovations and tools coming.

Cornett showed “search a sketch,” a novel mobile search tool enabling users to draw custom area on a smartphone map and search within that area. Similar to Yahoo Local’s radius search online, it was an impressive. He explained philosophically that Yahoo would take a very different approach in mobile vs its competitors who are largely reproducing the PC experience on smartphones.

David Pann, Search Advertising VP ended the formal part of the meeting with discussion of some improvements on that side of the house, including a Google campaign import tool.

At the end of the presentations, I had the feeling that “we’ve heard much of this before” and got the sense that much of the audience remained unconvinced that Yahoo would continue to be competitive in search. However the Q&A portion of the discussion and the informal lunch discussions thereafter were both useful and more convincing.

Candidly Seth acknowledged the search share losses that comScore and Hitwise have reported. He attributed some of those losses to toolbar and default search deals that Yahoo no longer had or that others had aggressively grabbed away. Overall I was impressed with the candor and frankness of Yahoo’s discussion in response to question after question that in one way or another was highly skeptical of Yahoo’s prospects — including mine.

I said: “many in this audience want Yahoo to be competitive and succeed but remain skeptical that you can.” Larry Cornett responded that he was at Apple during the “dark time” before the return of Steve Jobs, when critics were calling for the company to shut down. He likened the media perceptions of Yahoo Search now to that period in Apple’s history and promised that Yahoo Search would “be back.” He added that many of Yahoo’s innovations were being freely copied by its competitors but in more superficial ways.

At the end of Q&A other Yahoo research scientists were introduced:

• Yoelle Maarek, Senior Director at Yahoo Research in Israel
• Ricardo Baeza-Yates, VP of Research for Europe and Latin America
• Preston McAfee, VP and Research Fellow at Yahoo Research in Burbank, CA

I spoke at some length with Maarek, who was previously at Google. She contrasted the environments and spoke about why she left Google and came to Yahoo. She spoke about greater openness and “humility” at Yahoo and of the opportunity to work with the Yahoo Research team, which she greatly respected. I also spoke a bit with Tim Mayer, who remains at Yahoo and expressed confidence in the product and the team.

I came away from the Q&A and these more informal discussions much more “convinced” than during the formal presentations that Yahoo would in fact continue to innovate and would retain and perhaps gain new search talent. The challenge is resources and mindshare, given how Google dominates the news with each (almost daily) product announcement.

But it would appear that Yahoo will continue to do interesting and innovative things in search — and the industry wants and needs them to.

Yahoo previewed these new features previously and Danny discussed them at some length in an earlier post. These changes are now in effect in the US, UK, France, Spain, Mexico, and India and will roll out more broadly to other markets in the near future. Overall there’s more visual appeal to the new format and the Search Monkey refinements are useful in most cases.

Below is an example search for London. Shortcuts remain at the top of results, now the center column:

If we click the “Yahoo Travel” filter in the left column, we get the following results without entering a new query:

Further down in the left column you can also refine by “weather” (not a Search Monkey filter, but a related “Search Assist” concept):

Search Assist continues to be a prominent feature of the overall search experience. And Search Pad gets more visibility:

The newly launched search format generally mirrors the new Yahoo homepage with its brand-centric left column:

With this new look and feel Yahoo Search moves somewhat closer to Bing, ironically. Bing has seen its market share slowly creep up and is probably a greater threat to its partner Yahoo than it is to Google.

In the relatively near future the actual results on Bing and Yahoo will be the same (assuming regulatory approval of the deal). Thus Yahoo will continue to need to make changes and upgrades to the interface and overall user experience to prevent a loss of usage and market share.

I haven’t systematically tested Yahoo’s new search against Google and Bing. However I generally like the new appearance as well as the Search Monkey features.

One thing that Yahoo should consider is a personalized left nav/column (a la the new home page) where users can select favorite Search Monkey filters (e.g., Yelp for local, Kayak for travel). Indeed, the more prominent appearance of Search Monkey in the new Yahoo Search may encourage more sites and publishers to participate in the program.

We hope that these changes are part of an ongoing series of upgrades to Yahoo Search, although many continue to believe that with the outsourcing of its back end and index to Microsoft, and exodus of key search engineers, that Yahoo has effectively given up on search. New Yahoo CMO Elisa Steele tried briefly to rebut that idea in her IAB MIXX keynote speech announcing the new Yahoo branding campaign.

Here’s more on the new search changes from the Yahoo Search Blog.

We have covered Yahoo SearchMonkey extensively here, for past articles see the Search Monkey category.

Postscript From Danny Sullivan: An important change is that those making use of Yahoo’s paid inclusion programs are now guaranteed rich results like those shown above, as long as they provide images in their data feeds.

Freed from the cost ($425 million reportedly) and ongoing demands of the back end, it would “innovate” around the interface and search user experience.

Many people in the industry were skeptical and  shrugged it off as something akin to wishful thinking or putting a brave face on Yahoo’s exit from the search business. (Danny wrote “A Search Eulogy for Yahoo” thereafter.) But Prabhakar Raghavan, in charge of Yahoo Labs, discussed on Friday, how the company might incorporate Twitter and real-time search into Yahoo results. He again made the case that Yahoo would continue to innovate and present a compelling search user experience.

“In terms of satisfying user intent, the hard work and in some sense the bigger growth opportunities for differentiation are not the back-end of crawling and indexing, but really surfacing and assembling content the right way to satisfy user intent,” he said.

Real Time search is an increasingly popular online activity where Yahoo’s approach to search could provide a compelling user experience, Raghavan said.

It’s theoretically possible but one of the practical challenges will be recruiting and retaining top search talent, as the NY Times points out:

Yahoo will lose some of its most talented engineers to Microsoft and as many as 400 employees through layoffs. The deal also undercuts years of investment around search technology. By selling the technological crown jewels, the company may lose some of its high-tech credibility among employees and others in Silicon Valley, as well as among customers.

Investors, who had been expected to cheer the deal, punished Yahoo for the terms of the transaction. On Thursday, the day the deal was announced, Yahoo’s stock declined roughly 12 percent — although it has largely recovered since then. Financial analysts and investors saw the deal terms as something of a fire sale. According to the Times article:

Microsoft offered $46 billion to buy all of Yahoo. Analysts estimate that the new deal — involving what many people saw as Yahoo’s most important asset — is worth only around $4 billion to $5 billion.

The deal was largely motivated by Ballmer’s persistence and Yahoo’s inability to compete with Google and Microsoft in search at the required levels of investment. There was also lots of additional pressure coming from the market. All of these things, among a couple of others, perhaps made the deal inevitable.

The deal isn’t done until regulators in the US and EU approve it. That process is not, as they say, a “slam dunk.” But let’s assume that it does go through. Yahoo can’t simply ignore search. It will still be forced by the market to pay attention to search volumes and monetization (RPS).

To that end, Yahoo will in fact need to invest in the user experience to maintain its position. If it doesn’t it will lose share to Google or Bing. If that starts happening Bartz and Yahoo will be under intense pressure. So the notion of “innovating around the user experience” isn’t simply an aspiration for Yahoo, it’s a necessity.

We’ve read about what this means for advertisers (more overall traffic from the combined audience, use of Microsoft adCenter for self-serve and Yahoo!’s sales force for premium) and for searchers (they likely won’t notice), but what does the deal means for those who create websites: publishers, web developers, and SEOs?

Web Developers

The hardest hit by this change will likely be developers. Over the last couple of years, Yahoo seems to have shifted its focus from innovating the search index to innovating its developer offerings: encouraging third-party development and creating a “developer ecosystem” for search.

Any developer options that don’t rely on the Yahoo search index may be unscathed. In particular, the non-search development tools and search-related offerings that are solely focused on the user interface may continue to be supported. While Bing will power Yahoo’s search engine, Yahoo will control their user interface and likely will try to continue to differentiate there. That’s means Search Monkey, which enables site owners to enhance how their results appear on Yahoo, is potentially safe.

Build Your Own Search Service (BOSS) likely won’t be so lucky. BOSS is built on the Yahoo index as its foundation. A company can build their own search engine using Yahoo’s underlying technology and differentiate via the user experience.  Essentially, that’s what Yahoo is planning to do now with Bing as their underlying technology. No more Yahoo index likely means no more BOSS. Yahoo all but concedes as much: “We can tell you that BOSS will remain live for the time being.” What does that mean for companies like hakia, OneRiot, Daylife, and Cluuz? And for that matter, all of the developers using BOSS who are now filling the Yahoo BOSS message boards with questions?

On the Yahoo developer blog, Yahoo commented that “For SearchMonkey and BOSS, we currently do not have anything concrete to tell you. Clearly, we’ll need to work with Microsoft to determine what makes the most sense for you and for us.” If BOSS’s future is left up to Microsoft, I have no doubt that future will involve migrating BOSS users to the Bing search API. In order to continue to support BOSS, Microsoft would have to completely recreate it to work with the Bing search infrastructure. Why would they do that when they can increase the audience of a product they already have? It’s possible they’ll add some of the unique BOSS features their search API (such as unlimited queries, ability to mash up the data with other sources, and ability to tweak ranking signals), but I wouldn’t hold my breath. The Yahoo BOSS team is just as in the dark as the developers wanting answers. From a message board post: “What specifically does it mean for BOSS? Honestly the team is still absorbing the implications and we just don’t know.”.

BOSS users could switch to Google’s Custom Search API, but it is more restrictive than Microsoft’s offering, and isn’t really well-suited as the foundation of a search engines or other commercial company. Several other companies offer web indices, such as CommonCrawl and Alexa, so perhaps they or a new company will take advantage Boss’s imminent demise and offer matching features.

Any Yahoo offerings that don’t rely on an underlying index, such as the Yahoo User Interface library are likely going to remain. Yahoo confirmed this in their blog post:

“We’ve also received questions about the future of Yahoo!’s other developer offerings, such as YUI, YQL , and Pipes. We wanted to let you know that today’s news does not affect these products. None of our other non-search developer products are affected.” [Emphasis mine.]

However, look for any search index-based offerings (such as the Maps API and Local API) to be deprecated in favor of the Bing equivalents once the deal goes through.

Search Engine Optimizers and Site Owners

What about those who are concerned with getting customer acquisitions through organic search? How will this change impact them? From a traffic perspective, take a look at how well you’re indexed and ranked in Bing. That’s how well you’ll be indexed and ranked in Yahoo. What do your titles and descriptions look like in the results in Bing? That’s how things will generally look in Yahoo. This might not be a bad thing for site owners, as over the last year, Yahoo’s search quality seems to have been declining to the point that I’ve been wondering if their engineering team had already begun to be phased out or least was spending a lot of time at the bar mourning the likely phase out.

Just as you don’t need to optimize separately for AOL since they use Google’s index, you won’t need to optimize for Yahoo since they’ll use Bing’s index. The exception to this may be in how Yahoo displays results. We’ll have to wait and see exactly what this means, but Bing has been trying to differentiate in display and it supposedly, Yahoo will continue to do that as well. This may mean, for instance:

• SearchMonkey will continue to be important as a way to stand out in the results.
• Hmm. I can’t really think of anything else.

My guess is that the robots-nocontent tag will no longer be supported, since Bing’s infrastructure doesn’t support it. The search engines have already come together to standardize their support of robots.txt and XML Sitemaps, so site owners shouldn’t worry about changing anything with those.

The bigger issue many SEOs are concerned about is Site Explorer. Site Explorer is one of the more reliable tools for competitive backlink research. You can see a substantial list of links to any site, generally ordered according to value. That’s useful stuff! Both Google and Bing Webmaster Tools provide backlink data, but only for your own sites. Yahoo will be unable to maintain Site Explorer without a search index of their own. Will Bing take it over? Well, it could add the feature to its Webmaster Tools, but Microsoft has historically been moving the other direction. They removed the ability to query their index for link data with the link: operator in 2007 and have never brought it back for competitive research.

Microsoft likely won’t be motivated to add a feature that they specifically chose to remove. And it’s not trivial to build the code to query for competitive links and store the data. Believe me, I know. I managed the process for adding non-competitive backlink data to Google Webmaster Tools. As with the potential end to BOSS, the potential end to Site Explorer opens up new opportunities for third-parties. In fact, the same companies who build a web index could provide competitive link data. Currently, SEOmoz  provides Linkscape, which offers some similar features. (Speaking of SEOmoz, Rand Fishkin posted yesterday about the SEO impact of this deal.) Majestic SEO and Exalead provides link data as well.

More generally, will Microsoft step up its efforts with webmaster relationships? Yahoo used to have a fairly significant presence in the community. In addition to Site Explorer, they were a constant at conferences and participated in online discussions. That participation has declined lately, coinciding with the decline in search quality. Microsoft seemed to be rallying with its webmaster relationships with the launch of the Webmaster Center in November 2007. But Microsoft hasn’t updated the Webmaster Center with new features since August 2008. (A minor release in November didn’t add  new features).

Microsoft didn’t respond to my questions about their current and future resource investment in this area. They did recently release a rudimentary SEO Toolkit, although it requires Windows Vista and IIS 7.0 to run.

And what about paid inclusion? Yahoo has long offered Search Submit Pro, which essentially enables sites to pay to be included in the organic listings. Microsoft doesn’t offer a similar product and while it’s certainly possible that Microsoft will add this product to their offerings, paid inclusion is quite a substantial shift in overall approach to organic search. It’s less about the ability to implement the technology and more about belief around what constitutes an “organic” index. Danny Sullivan asked about paid inclusion at the announcement press conference. Carol Bartz, Yahoo CEO replied, ” Paid inclusion, we’ll decide on that later.” But it would be difficult for Yahoo to continue the program on its own, as Yahoo will no longer have control over what pages are included in the search index.

In the end, it’s about the traffic

The big question is will this partnership significantly change market share percentages? Depending on whose numbers you use, Google has either 65% or 74% share in the US (more in some European countries). That puts the combined Yahoo/Microsoft share at 28% or 25.5%. That’s substantial traffic, sure, and worth paying attention to. But what will the share look like in three years once the deal is done and we barely remember Yahoo ever had its own index? My guess is pretty similar to how it looks now. Except Google will probably have slightly higher share. I just don’t see anything game changing here that will cause a mass exodus from the status quo. But I’ve been wrong before. What I do know is that site owners who have ignored how their sites were doing in Bing until now do to low traffic numbers will likely start paying a lot more attention.

The existence of a forthcoming Yahoo homepage redesign has been known for months and screenshots of earlier versions have been seen previously in several places online. However, the new Yahoo homepage, which launches around 4:30 Eastern today, turns out to be both interesting and practical. The embargo was broken yesterday so immediately everyone rushed out their stories and there’s already been considerable coverage.

Here’s the Yahoo homepage as it exists this morning:

Below is the new homepage. It will not immediately replace the one above but be an opt-in choice for those interested in the near term:

Overall the page is less cluttered, there are fewer buttons and modules and search is somewhat more prominent. In the current Yahoo homepage search seems to exist outside the main visual field, whereas in the new design it’s more “visible” and integrated into the rest of the page.

The left column goes from being a static list of Yahoo properties to a customizable menu that effectively turns the homepage into an RSS reader or “dashboard.” Borrowing from MyYahoo (which will continue on as it is), Yahoo allows users to add or remove widgets or applications. Those include third party sites.

Yahoo has created a range of these widgets for launch, but users can themselves add any site with RSS feed capability. And third party developers will soon be able to add their own applications to the gallery. In this way it conceptually brings together the homepage and what Yahoo is doing with SearchMonkey.

Typically users don’t do much customization. I raised this question with Yahoo SVP Tapan Bhat who was presenting the new homepage. He agreed and said that Yahoo will recommend widgets to users based on their browsing and click-stream behavior. The process of adding or removing “applications” is simple and basically involves a single click.

As users mouse over the individual “My Favorites” applications a window opens that allows them to see the site or content. Here’s the official Yahoo screenshot with Facebook as an example.

Users can update Facebook (or Twitter) from this window. On the right side of the window is a new, contextually relevant ad unit that can be targeted to the content of the page. One example Yahoo gave on the call was for a movie opening targeting users looking up movie showtimes from a Yahoo movies application.

I haven’t been able to use the new homepage but it appears Yahoo has done a nice job balancing “push” and “pull,” adding simple but useful customization without diminishing the broad reach that the page offers.

Users will also later be able to adjust the news that they see on their homepage on a sliding fun-to-serious scale:

In terms of search, Yahoo will start “bucket testing” (random testing) a new look and feel for search results that conceptually mirrors what it’s doing on the home page. Below are screens I captured from the presentation. In the left column are (Search Monkey) widgets that effectively become search filters. The second example below shows the same search only showing results from YouTube:

Similar to the homepage favorites, the benefits of this approach rely on people customizing search and adding filters. However it’s pretty interesting on multiple levels. This search page will not be widely available today when the new homepage launches because it’s being selectively tested. In fact it’s possible it may never roll out.

How all this may or may not play with Microsoft if or when a search and display ads deal happens is uncertain. The question was asked on the call about how Bing might tie into this page. Bhat, as one would expect, demurred and declined to say anything.

Finally, the new homepage will make its way to mobile fairly soon. The recently launched mobile portal was ahead of the PC site in terms of customization. But the changes should make it simpler and more useful still.

The new homepage will roll out first in the US, followed by the UK, India, France and other countries thereafter.

Now Yahoo has announced support for several additional opportunties for automated enhanced listings.  You can use DataRSS XML to send a feed directly to Yahoo or use RDFa or microformat markup on your pages for content such as products, events, and news. In addition, Yahoo can now use a Google Base feed to create enhanced listings. Simply submit your Google Base feed through Site Explorer. As I mentioned in my analysis of Common Tag, these moves seem to be motivated at least in part by Yahoo’s quest to increase the reach of BOSS. Yahoo’s blog post notes that when you submit your Google Base feed, Yahoo converts it to DataRSS and makes it available in BOSS so it can be used for third-party search engine development.

While none of the search engines are using structured data for crawling, indexing, or ranking, it’s still likely worthwhile to include this markup for data like events, news, and reviews to take advantage of the opportunity to have a search result that stands out in Yahoo and Google (and potentially in Bing as well).

Yahoo! says that Common Tag makes “web content more discoverable” and enables the community to “create more useful applications for aggregating, searching, and browsing the web.” Their blog post mentions that they want to accelerate the structuring of the web, which aligns with their SearchMonkey launch last year, which they said was, in part, an attempt to encourage the use of structured data on the web. This brings to mind a few questions.

Why did the web need a new semantic standard? The Common Tag blog explains:

“Semantic web promises havens. The promise of machines understanding the data and acting on it semi-autonomously. But how do we get there? Not many are willing to put in a lot of work into publishing data in new formats. For the benefit that might or might not come years after enormous resources will be spent?

So why not start with something easy, say tagging? Marking up text with exactly defined concepts. “

OK, maybe “explains” isn’t the right word.

This question really goes to the heart of what’s curious about Common Tag. Yahoo called it a “new semantic tagging format” in its blog post, but when we asked them why the web needed something new, they clarified that it’s an RDFa vocabulary, not something made from whole cloth. The Common Tag About page also implies that this is really just part of the standards that all the major search engines have joined together to support. “In addition, search engines like Yahoo and Google have begun reading RDFa—the markup standard used by the Common Tag format—to acquire richer information about sites that use it… Google’s new Rich Snippets feature uses the information to apply similar enhancements to Google search results.”

In truth, none of the major search engines are using semantic markup in web search and Google is using existing standards (microformats and RDFa) to display enhanced listings. Both Google and Yahoo have told me that they could use metadata in web search in the future, if it proves to be useful and they can safeguard against spamming. So far, this hasn’t happened.

Yahoo did clarify to me that Common Tag is something they’re participating in as means to cultivate the structured data community, not something they’ve come up on their own and are trying to get the community to adopt. RDFa provides a structure from which you can create vocabularies and several companies who were using RDFa were interested in creating a tagging vocabulary. Since these companies used SearchMonkey as an application for their metadata, they asked Yahoo to help create and promote  this new vocabulary.

So, how does it work? Common Tag is intended to be a common tagging format to standardize tagging of concepts. According to the commontag.org site, as “publishers, developers, and end users” join in support for this format, “more content related to a specific concept will be discoverable through a single tag.” Now, for instance, the concept New York City may be tagged with “nyc”, “new_york_city”, and “newyork”. You can adding this tagging markup to your pages manually, or you can use infrastructure such as that provided by founding company Zemanta. And you can eliminate the problem of multiple tags for the same concept by using data from a participating database, such as Freebase (also a founding company). You can then use this structured data in an application such as Yahoo! SearchMonkey.

For instance, the Common Tag documentation uses the following example of using the Freebase database to tag a page as being about U2:

<body xmlns:ctag="http://commontag.org/ns#" rel="ctag:tagged">
    <span typeof="ctag:Tag" rel="ctag:means"
         resource="http://rdf.freebase.com/ns/en.u2"/>
</body>

You can also do more complicated tagging, such as of external resources, sections of your web pages, and concepts within your content. For instance, you can identify the paragraphs of text on the page as follows:

<p id="first">Everyone loves Buffy the Vampire Slayer. </p>
<p id="second">Amber Benson was awesome in it.  </p>

And then create tags for those paragraphs:

<div xmlns:ctag="http://commontag.org/ns#"
         about="#second" rel="ctag:tagged">
    <span typeof="ctag:Tag"
         rel="ctag:means"
         resource="http://dbpedia.org/resource/Amber_Benson"/>
</div>

Why is Yahoo! so hell-bent on covering the web with structure? If Yahoo! found structured data made the web easier to crawl and their search results more relevant, I could see the push. But Yahoo! doesn’t use any of the semantic formats they’re encouraging in web search. They already were encouraging hCard, hCalendar, hReview, hAtom, XFN, Dublin Core, Creative Commons, FOAF, GeoRSS, MediaRSS, RDFa, and OpenSearch. Why do they need web developers to start using yet another format when they haven’t yet figured out how to use all of those others in their core search engine? Sure, they are involved in Common Tag in order to support the structured data community they’ve been aiming to accelerate, but why is that so important to them?

Since Yahoo isn’t encouraging the use of semantic markup to help them get an edge in search, it seems they must be instead looking to increase adoption of SearchMonkey and BOSS, where these formats are used.They seemingly have diverted the energy they used to spend to help improve Yahoo’s search index via tools such as Site Explorer to working to raise adoption of BOSS. The last Site Explorer update was in August 2008, and that was simply a UI change. No new features were launched. For new features, you have to go all the way back to August 2007, for dynamic URL rewriting.

Even Yahoo CEO Carol Bartz isn’t talking about focusing their consumer search engine as a core offering, but rather something that’s convenient for Yahoo users who are already on the site for some other reason (emphasis mine):

Listen, Google has the search brand, there’s no doubt about it …. [Yahoo Search] is really designed for people that are on our sites and find something interesting, they want to look farther and they go to Yahoo Search.

It’s enough to make one wonder if Yahoo is quietly abandoning its consumer search engine in favor of accelerating new third-party search engines through BOSS. If you can’t beat ‘em, help their enemies attack them on all fronts, as the old saying goes.

Didn’t the search engines already try using meta tags? The idea of using meta data to tag web pages in order to describe them to search engines isn’t new, of course. The meta keywords tag has been around since at least 1995. And it’s easier to adopt than Common Tag. That U2 example? The meta keywords tag would only require this:

<meta name="keywords" content="U2">

Indeed, Yahoo supported the meta keywords tag initially (and to some extent, still does), but when Google launched, they did not. It was too easy for site owners to stuff that tag with anything they wanted, rather than the true focus of the page. Search engines use smarter methods (starting with the content on the page and how external sites link to it) for determining relevance. Could Common Tag have the same downfall? After all, as the documentation explains “you can create as many Tags as necessary to describe the contents of a document.”

Not only does Common Tag seem to replicate the purpose of the meta keywords tag, it seems to also replicate Delicious-style tagging and external anchor text. From the site:

“Common Tags are not only useful for identifying the concepts covered in your content, but if you reference content elsewhere on the web, Common Tags can be used to indicate the concepts covered in that external content as well. This is useful for better describing and organizing the content of external resources from within your own content. For example, you could use Common Tags to publish bookmarks to identify the concepts described by a link, or you could use them categorize image collections stored elsewhere on the web.”

A microformat already exists for a similar purpose as well.  rel=”tag” is intended to tag content, such as web pages or portions of them.

Anchor text is an established method for search engines to determine how others describe an external resource. As for tags, the study Can Social Bookmarking Improve Web Search, presented at the First ACM International Conference on Web Search and Data Mining (Stanford) analyzed 40 million Delicious tags and found that anchor text was a better signal for web search relevance. Part of the problem was scale of adoption. A lot of people have to adopt this new tagging method for it to be worthwhile to use across the web. And if Delicious tags don’t have the scale, how long will it take for Common Tag to?

When I asked Yahoo about it this, they acknowledge that it may not be something that’s adopted web-wide. Rather, it’s a format of interest to a particular group of developers who have needs beyond that which is available through means such as the meta keywords tag and rel=”tag”.

Why would anyone implement this? It seems like a lot of work. You can tag content now using methods like anchor text and well, tags, such as those available through most blogging platforms and bookmarking sites like Delicious. If content management systems and other content creation platforms such as blogging systems incorporate this structure (for instance, by automatically using the tags labeling a blog post), we might see some adoption, but this wouldn’t eliminate the issue of multiple tags for one concept. (Zemanta, one of the founding companies for Common Tag provides plugins for blogging platforms to insert Common Tagging.) And WordPress strips out RDFa by default.

The answer is that web developers will use this structure, just as they’ll use any other structure, if it’s valuable for what they’re building. And what applications are ultimately made possible by this format remain to be seen.

The big idea (now familiar) is moving beyond “10 blue links” (popularized as a criticism of search by former Ask CEO Jim Lanzone) to a “web of objects.” The “web of objects” presented by Yahoo is a better representation of the “real world” in search results. In other words: more closely aligning user intent with search results and mapping those to real-world tasks. Conceptually I agree with this approach, even though it doesn’t cover every search use case.

Prabhakar Raghavan, Head of Yahoo! Labs and Yahoo! Search Strategy, explained that this concept was partly drawn from Yahoo’s experience with mobile search. The idea is that people are ultimately trying to actually do things in places and that there’s a larger context to user intent and search behavior. “We’re moving toward surfacing real-world objects rather than documents,” said Raghavan.

Raghavan added that Yahoo is not going to be concerned about index size going forward. Rather Yahoo will be building these composite bundles of structured data. As a practical matter, these web objects are manifested in the form of multi-media content and images (Shortcuts). The broader objective is to provide more context and “answers” to minimize links and clicking back and forth.

Larry Cornett, Vice President, Consumer Products, Yahoo! Search, said that Yahoo was bucket testing different content presentations. In one experimental page he showed a query for “Paris” in which only images were presented.

The context in which all this has perhaps the most obvious and immediate impact is in mobile, as mentioned. Marc Davis, Chief Scientist, Yahoo Mobile, provided a range of mobile search examples from movies to restaurants and travel. These examples were fairly compelling in terms of how oneSearch delivered a range of information and often actionable content within a single click.

As part of his portion of the presentation, Larry Cornett walked the group through the ways that many of Yahoo’s existing technologies are supporting the new conceptual approach:

• Greater insight into user intent (e.g., Search Assist)
• Developing a web of objects (reflected in Shortcuts and non-textual content)
• Open initiatives that tap third parties and the crowd for structured data (i.e., Search Monkey, BOSS)

He also announced that it was the 1st anniversary of Search Monkey and threw out a bunch of stats and milestones:

• It’s in 23 markets globally
• 70 million enhanced Search Monkey results viewed daily
• Search Monkey results see as much as 15% improvement in CTRs
• 15K developers are using it
• Provides partner branding – through video, music, documents, flash games on the SERP

There’s something profound in Yahoo’s concept of “web objects” and the relationship between search and the real world. It’s about building more context into and around search. The challenge for Yahoo is taking this elegant idea or concept and actually making it real for people.

Here’s more from the Yahoo Search Blog. TechCrunch live blogged the discussion.