Yahoo continues to add new data to its popular Site Explorer tool and, for reasons unknown, doesn’t tell anyone about it. Today, Barry Schwartz reported on Search Engine Roundtable about several new additions to Site Explorer’s data:

• Key Terms
• Delicious Activities
• Top Delicious Tags
• SearchMonkey Objects

Barry posted this screenshot showing the new features as they related to Search Engine Roundtable:

It was just a month ago that Yahoo expanded Site Explorer with a new “Top Queries/Top URLs” feature. Webmasters and search marketers still wonder, though, what Site Explorer’s fate will be if the Yahoo/Microsoft search deal gets approved. Yahoo hasn’t commented on our questions about that.

Postscript by Barry Schwartz: Yahoo has now announced it on the Yahoo Search Blog.

I noticed this morning that Yahoo has added a new feature named “Top Queries” to Site Explorer. I have not seen any announcement from Yahoo on this addition, but it seems pretty new. The tool shows you the top ten queries by day, week or month. It also shows you your top ten URLs by day, week or month. Yahoo classifies “top” by the number of impressions you received for those queries or URLs, but it also shows you the clicks.

The last time we reported about new Site Explorer features was over a year ago. Back then, the action bar had less features, the one on the left is the old one and the one on the right is the new one:

Note, I did crop the bottom of the new one off, simply because that matches the old one.

Here is a closer look at these reports:

By now, everyone has read all about the news that Yahoo is replacing its search index with Microsoft’s Bing. In a way, it’s a great story of complete reversal, as in 2002, Microsoft didn’t have its own index and instead used Inktomi. Late that year, Yahoo! acquired Inktomi, which spurred Microsoft to start building its own search index to avoid having a search supplier owned by a major competitor. Now Yahoo is ditching its index (including all of the technology it acquired with Inktomi) to use the very index it motivated Microsoft to build.

We’ve read about what this means for advertisers (more overall traffic from the combined audience, use of Microsoft adCenter for self-serve and Yahoo!’s sales force for premium) and for searchers (they likely won’t notice), but what does the deal means for those who create websites: publishers, web developers, and SEOs?

Web Developers

The hardest hit by this change will likely be developers. Over the last couple of years, Yahoo seems to have shifted its focus from innovating the search index to innovating its developer offerings: encouraging third-party development and creating a “developer ecosystem” for search.

Any developer options that don’t rely on the Yahoo search index may be unscathed. In particular, the non-search development tools and search-related offerings that are solely focused on the user interface may continue to be supported. While Bing will power Yahoo’s search engine, Yahoo will control their user interface and likely will try to continue to differentiate there. That’s means Search Monkey, which enables site owners to enhance how their results appear on Yahoo, is potentially safe.

Build Your Own Search Service (BOSS) likely won’t be so lucky. BOSS is built on the Yahoo index as its foundation. A company can build their own search engine using Yahoo’s underlying technology and differentiate via the user experience.  Essentially, that’s what Yahoo is planning to do now with Bing as their underlying technology. No more Yahoo index likely means no more BOSS. Yahoo all but concedes as much: “We can tell you that BOSS will remain live for the time being.” What does that mean for companies like hakia, OneRiot, Daylife, and Cluuz? And for that matter, all of the developers using BOSS who are now filling the Yahoo BOSS message boards with questions?

On the Yahoo developer blog, Yahoo commented that “For SearchMonkey and BOSS, we currently do not have anything concrete to tell you. Clearly, we’ll need to work with Microsoft to determine what makes the most sense for you and for us.” If BOSS’s future is left up to Microsoft, I have no doubt that future will involve migrating BOSS users to the Bing search API. In order to continue to support BOSS, Microsoft would have to completely recreate it to work with the Bing search infrastructure. Why would they do that when they can increase the audience of a product they already have? It’s possible they’ll add some of the unique BOSS features their search API (such as unlimited queries, ability to mash up the data with other sources, and ability to tweak ranking signals), but I wouldn’t hold my breath. The Yahoo BOSS team is just as in the dark as the developers wanting answers. From a message board post: “What specifically does it mean for BOSS? Honestly the team is still absorbing the implications and we just don’t know.”.

BOSS users could switch to Google’s Custom Search API, but it is more restrictive than Microsoft’s offering, and isn’t really well-suited as the foundation of a search engines or other commercial company. Several other companies offer web indices, such as CommonCrawl and Alexa, so perhaps they or a new company will take advantage Boss’s imminent demise and offer matching features.

Any Yahoo offerings that don’t rely on an underlying index, such as the Yahoo User Interface library are likely going to remain. Yahoo confirmed this in their blog post:

“We’ve also received questions about the future of Yahoo!’s other developer offerings, such as YUI, YQL , and Pipes. We wanted to let you know that today’s news does not affect these products. None of our other non-search developer products are affected.” [Emphasis mine.]

However, look for any search index-based offerings (such as the Maps API and Local API) to be deprecated in favor of the Bing equivalents once the deal goes through.

Search Engine Optimizers and Site Owners

What about those who are concerned with getting customer acquisitions through organic search? How will this change impact them? From a traffic perspective, take a look at how well you’re indexed and ranked in Bing. That’s how well you’ll be indexed and ranked in Yahoo. What do your titles and descriptions look like in the results in Bing? That’s how things will generally look in Yahoo. This might not be a bad thing for site owners, as over the last year, Yahoo’s search quality seems to have been declining to the point that I’ve been wondering if their engineering team had already begun to be phased out or least was spending a lot of time at the bar mourning the likely phase out.

Just as you don’t need to optimize separately for AOL since they use Google’s index, you won’t need to optimize for Yahoo since they’ll use Bing’s index. The exception to this may be in how Yahoo displays results. We’ll have to wait and see exactly what this means, but Bing has been trying to differentiate in display and it supposedly, Yahoo will continue to do that as well. This may mean, for instance:

• SearchMonkey will continue to be important as a way to stand out in the results.
• Hmm. I can’t really think of anything else.

My guess is that the robots-nocontent tag will no longer be supported, since Bing’s infrastructure doesn’t support it. The search engines have already come together to standardize their support of robots.txt and XML Sitemaps, so site owners shouldn’t worry about changing anything with those.

The bigger issue many SEOs are concerned about is Site Explorer. Site Explorer is one of the more reliable tools for competitive backlink research. You can see a substantial list of links to any site, generally ordered according to value. That’s useful stuff! Both Google and Bing Webmaster Tools provide backlink data, but only for your own sites. Yahoo will be unable to maintain Site Explorer without a search index of their own. Will Bing take it over? Well, it could add the feature to its Webmaster Tools, but Microsoft has historically been moving the other direction. They removed the ability to query their index for link data with the link: operator in 2007 and have never brought it back for competitive research.

Microsoft likely won’t be motivated to add a feature that they specifically chose to remove. And it’s not trivial to build the code to query for competitive links and store the data. Believe me, I know. I managed the process for adding non-competitive backlink data to Google Webmaster Tools. As with the potential end to BOSS, the potential end to Site Explorer opens up new opportunities for third-parties. In fact, the same companies who build a web index could provide competitive link data. Currently, SEOmoz  provides Linkscape, which offers some similar features. (Speaking of SEOmoz, Rand Fishkin posted yesterday about the SEO impact of this deal.) Majestic SEO and Exalead provides link data as well.

More generally, will Microsoft step up its efforts with webmaster relationships? Yahoo used to have a fairly significant presence in the community. In addition to Site Explorer, they were a constant at conferences and participated in online discussions. That participation has declined lately, coinciding with the decline in search quality. Microsoft seemed to be rallying with its webmaster relationships with the launch of the Webmaster Center in November 2007. But Microsoft hasn’t updated the Webmaster Center with new features since August 2008. (A minor release in November didn’t add  new features).

Microsoft didn’t respond to my questions about their current and future resource investment in this area. They did recently release a rudimentary SEO Toolkit, although it requires Windows Vista and IIS 7.0 to run.

And what about paid inclusion? Yahoo has long offered Search Submit Pro, which essentially enables sites to pay to be included in the organic listings. Microsoft doesn’t offer a similar product and while it’s certainly possible that Microsoft will add this product to their offerings, paid inclusion is quite a substantial shift in overall approach to organic search. It’s less about the ability to implement the technology and more about belief around what constitutes an “organic” index. Danny Sullivan asked about paid inclusion at the announcement press conference. Carol Bartz, Yahoo CEO replied, ” Paid inclusion, we’ll decide on that later.” But it would be difficult for Yahoo to continue the program on its own, as Yahoo will no longer have control over what pages are included in the search index.

In the end, it’s about the traffic

The big question is will this partnership significantly change market share percentages? Depending on whose numbers you use, Google has either 65% or 74% share in the US (more in some European countries). That puts the combined Yahoo/Microsoft share at 28% or 25.5%. That’s substantial traffic, sure, and worth paying attention to. But what will the share look like in three years once the deal is done and we barely remember Yahoo ever had its own index? My guess is pretty similar to how it looks now. Except Google will probably have slightly higher share. I just don’t see anything game changing here that will cause a mass exodus from the status quo. But I’ve been wrong before. What I do know is that site owners who have ignored how their sites were doing in Bing until now do to low traffic numbers will likely start paying a lot more attention.

The Yahoo Search Blog announced that they are testing out a new design for Site Explorer plus offering a variety of new stats about web sites.

The new design is available for all to see at a special, beta URL located at http://siteexplorer.search.yahoo.com/new.

As part of the new design, Yahoo added a site summary page that shows additional stats about registered/authenticated sites, including:

+ Site URL + Number of pages known + Number of pages crawled + Number of host on this domain + Number of inlinks + Number of inlink domains + Number of outlinks + Number of outlink domains

Also, the number of Dynamic URL Rewriting rules has been increased from 3 to 10.

Here are two screen shots of the new interface: Overview Page:

Sidebar Expanded:

Last year on Christmas Day, Barry reported over at Search Engine Roundtable how Adam Lasnik and I spent a bit of time answering questions on the Google Webmaster Central forums. This year, Google Webmaster Trends Analyst John Mueller keeps up the tradition.

In the spirit of the season, I thought it might be fun to take a look at the webmaster forums offered by the various search engines: a Christmas gift of search engine support.

Google The Google Webmaster Central forums began as a Sitemaps-specific discussion area (originally launched in June 2005 in English and German), and in August 2006 it was expanded as part of the launch of Webmaster Central to contain five topic areas about all things webmaster. Groups now exist in 22 languages, at least 12 of which are monitored by Googlers.

How’s the current level of participation? A quick glance shows John Mueller posting yesterday as well as today, and several Googlers posting within the last week. Webmasters are discussing all kinds of topics — from questions about Webmaster Tools specifically, to general ranking issues, to technical web site topics.

Yahoo! Yahoo! actively monitors their Site Explorer Suggestion Board and provides a response or status for many threads. The current board has 10 months of active topics and has a Digg-like rating system. This system replaced the previous forum, which has posts dating back to August of 2006.

A quick check of the topics shows that Yahoo! employees have been posting over the last week and topics tend to focus on Site Explorer and indexing issues.

Microsoft Live Search Live Search launched their current support forum as part of the Live Search Webmaster Center, which launched in public beta in November.

A quick check of the discussions shows that the Live Search team has been posting in the last week and the questions are (you guessed it) primarily about the Live Search Webmaster Tools and indexing/ranking issues.

All three have active blogs as well, where team members report the latest news, answer questions, and provide details about the issues webmasters are talking about.

• Google Webmaster Central Blog
• Yahoo! Search Blog (Site Explorer category)
• Live Search Webmaster Center Blog

All this activity is great news for webmasters. All three major search engines are actively monitoring discussion forums just for site owners, and all have multiple people not only taking note of issues, but personally answering questions (and offering homemade eggnog).

No word on whether you have to bring your own rum.

We reported earlier about Yahoo Site Explorer not showing all link data. Yahoo has just posted an update for us on that topic.

Yahoo says to “disregard any counts for inlinks reported by Site Explorer from October 11 through next week.” Yahoo is rolling out fixes for the tool over the weekend and throughout next week.

Postscript: On November 6th, Yahoo announced they have finally fixed the issue.

Yahoo Site Explorer Showing Different Counts For Registered vs. Non-Registered Users from the Search Engine Roundtable is where I reported that Yahoo Site Explorer seems to be showing a lot more linkage data to a logged in user who authenticated their site in Site Explorer as opposed to a user who is not registered with Yahoo.

For example, when viewing the linkage data for the Search Engine Roundtable as a guest, I see only 57,146 inlinks to the home page, but when I login under the username that authenticated that site, I see 216,880 inlinks to the home page.

Do you think Yahoo is showing only a sampling of data to non-registered users? Do you think that Yahoo will show complete linkage data to only users who have authenticated their sites?

I created a Sphinn thread for further discussion.

The Yahoo Search Blog announced a new Site Explorer feature that enables webmasters to inform Yahoo about dynamic URLs that might be creating duplicate content issues or crawl barriers.

The new feature is named “Dynamic URL Rewriting,” and it allows webmasters to view potential duplicate content issues and have Yahoo rewrite those URLs at the index level. Yahoo allows you to define which URL is the primary, and the remaining URLs will be dynamically rewritten to that primary URL.

One of the most discussed advanced topics in SEO is duplicate content. Duplicate content can often be generated by accident by having automated dynamic URLs. How so? The URLs can append characters to the URLs such as session ids, tracking parameters, format modifiers and other parameters that do not change the content of the page but yet create a new unique looking URL. Savvy SEOs detect these duplicate URLs and use rewriting technology to inform the search engine of the primary URL. But sometimes webmasters forget.

This new tool will enable webmasters to easily identify these URLs and classify the primary versions of the URLs.

For example, say you set up a tracking URL for a banner ad such as searchengineland.com/?ad_id=1234, which is basically a duplicate URL for searchengineland.com. You can tell Yahoo that anything that conforms to the “ad_id” parameter following the ? will by a duplicate URL to the primary and Yahoo will rewrite the URL to default to the primary.

Here is a screen capture of the tool:

As you can see, Yahoo gives you two options with the tool

(1) Remove these parameters from the URLs, such as in case of session ids, you could ask to remove ’sid’ from URLs (2) Use a default value for the parameter, for example you could set the ’src’ parameter to be ‘yhoo_srch’

It currently seems you can apply up to three parameters for the rewriting or removal in this tool.

Yahoo has done a great job explaining how it works at the Yahoo Search Blog and over at Yahoo Site Explorer help.

Last November, Google, Microsoft and Yahoo united to support sitemaps, a standardized method of submitting web pages through feeds to the search engines. Today, the three are now joined by Ask.com in supporting the system and an extension of it called autodiscovery. This is where the major search engines will automatically locate your sitemaps file if the location is listed in a robots.txt file. Announcements are up from Google and Ask now Yahoo and Microsoft.

Information on how to create sitemaps files can be found at the Sitemaps.org site. Aside from the sitemaps XML formal, you can also provide RSS 2.0 or Atom 0.3 or 1.0 feeds. That’s handy for those with blogs that already generate these feeds.

Sitemaps XML files too complicated? Don’t run a blog? Note that the site has newly expanded information on how you can submit a simple list of URLs in a text file.

In the past, if you created a sitemaps file, you then had to manually tell the search engines where to find it. With today’s announcement, search engines will check your robots.txt file for a link to a sitemaps file, then get the file from that location. This is a big plus because all the major search engines regularly check robots.txt files as part of their ordinary crawling.

To add the location, just put a line like this anywhere in your robots.txt file:

Sitemap: LOCATION-OF-SITEMAPS-FILE

Replace the LOCATION-OF-SITEMAPS-FILE with the actual location. For example, if you ran a site at mywonderfulsite.com and had a sitemaps file called allmypages.xml in your top level, the reference would be like this:

Sitemap: http://mywonderfulsite.com/allmypages.xml

Have more than one sitemaps file? Ideally, you’d create a special "sitemaps index" file that links to all of them, then put a link to the sitemaps index file in your robots.txt file. If that sounds like too much work, you can have more than one sitemaps URL listed in the robots.txt file.

Aside from autodiscovery, you can also ping Google and Yahoo with the location of your file. The Sitemaps.org site has more instructions on this in general. For specifics:

• Google: See here. Note that this pinging is different than the pinging Google also supports for blog search.
   
• Yahoo: See here and here. Unlike Google, the same pinging system is used for both web and blog search, to my understanding.

Both Google and Yahoo also allow you to manually submit sitemaps files. In both cases, doing this via their Google Webmaster Central or Yahoo Site Explorer systems gives you access to specialized monitoring and reporting tools or information on how they crawl you.

For more about these tools, or how each individual search engine handles sitemaps files, please see the links below:

• Google: Google Webmaster Central or sitemaps info.
• Yahoo: Yahoo Site Explorer or sitemaps info.
• Microsoft Live: submit page or submit help info
• Ask: webmaster help info

Keep in mind that Microsoft and Ask are still lacking references to sitemaps information, but I expect this will change over time.

For related coverage, see here and here on Techmeme.

Site Explorer Matures a Bit More and Accepts Mobile Feeds from the Yahoo Search Blog covers how the Yahoo Site Explorer tool now accepts mobile submissions and feeds, gains the ability to input URLs for deletion and remove more than five at a time (see our Yahoo Site Explorer Adds Features: Delete URL From Index article for more about the remove feature in general), plus a new "Report Spam" button that’s been added.