Where We Came From

Some might believe that search engines hate SEOs, hate publishers and have done little over the years to help them. They are mistaken, either choosing to deliberately ignore the gains or, more likely, are simply unaware of how far things have come.

When I first started writing about SEO issues nearly 16 years ago, in 1996, we had little publisher support beyond add URL forms. Today, we have entire toolsets like Google Webmaster Central and Bing Webmaster Tools, along with standalone features and options, which allow and provide:

• Ability to submit & validate XML sitemaps
• Ability to view crawling & indexing errors
• Ability to create “rich” listings & manage sitelinks
• Ability to migrate a domain
• Ability to indicate a canonical URL or preferred domain
• Ability to set crawl rates
• Ability to manage URL parameters
• Ability to view detailed linkage information to your site
• Ability to view keywords used to reach your site
• Notifications of malware or spam issues with your site

There’s even more beyond what I’ve listed above. The support publishers enjoy today was simply unimaginable to many veteran SEOs who were working in the space a decade ago.

The advancement has been welcomed. It has helped publishers better manage their placement in those important venues of the web, the search engines. It has helped search engines with errors and problems that would hurt their usability and relevancy.

That’s why 2011 was so alarming to me. After years of moving forward, the search engines took a big step back.

The Loss Of Link Data

One of the most important ways that search engines determine the relevancy of a web page is through link analysis. This means examining who links to a page and what the text of the link — the anchor text — says about the page.

However, for years Google has deliberately suppressed the ability for outsiders to see what links tell it about any particular page. Want to know why THAT result shows up for Santorum? Why Google was returning THAT result for “define English person” searches? Sorry.

Google won’t help you understand how links have caused these things. It refuses to show all the links to a particular page, or the words used within those links to describe a page, unless you are the page’s owner.

Why? Google’s rationale has been that providing this information would make it harder for it to fight spam. Potentially, bad actors might figure out some killer linking strategy by using Google’s own link reporting against it.

It’s a poor argument. Despite withholding link data, it’s painfully easy to demonstrate how sites can gain good rankings in Google for competitive terms such as “SEO” itself by simply dropping links into forums, onto client pages or into blog templates.

Given this, it’s hard to understand what Google thinks it’s really protecting by concealing the data. But until 2011, there was an easy alternative. Publishers and others could turn to Google-rival Yahoo to discover how people might be linking to a page.

Goodbye Yahoo Site Explorer

Yahoo launched its “Yahoo Site Explorer” back in September 2005, both as part as a publicity push to win people away from Google and to provide information to publishers. The tool allowed anyone to see what link data Yahoo had about any page in its listings.

Today, Yahoo still supposedly wants to win people away from Google. But because Yahoo’s web search results are now powered by Bing, Yahoo has little reason to provide tools to support publishers. That’s effectively Bing’s problem now.

Yahoo closed Yahoo Site Explorer at the end of last November, saying as it still does on the site now:

Yahoo! Search has merged Site Explorer into Bing Webmaster Tools. Webmasters should now be using the Bing Webmaster Tools to ensure that their websites continue to get high quality organic search traffic from Bing and Yahoo!.

That’s not true. Yahoo Site Explorer was not merged into Bing Webmaster Tools. It was simply closed. Bing Webmaster Tools doesn’t provide the ability to check on the backlinks to any page in the way that Yahoo Site Explorer allowed.

The closure supposedly came after Yahoo “listened to your feedback” about what publishers wanted, as it posted earlier this year. I don’t know what feedback Yahoo was hearing, but what I’ve heard has been people desperately pleading with Yahoo or Bing to maintain the same exact features that Yahoo Site Explorer provided — and pleading for well over a year.

Yahoo-Bing Deal Has Reduced Competition & Features

When the US Department Of Justice granted its approval for Yahoo to partner with Microsoft, that was supposed to ensure that the search space stayed competitive. From what the Department Of Justice said in 2010:

After a thorough review of the evidence, the division has determined that the proposed transaction is not likely to substantially lessen competition in the United States, and therefore is not likely to harm the users of Internet search, paid search advertisers, Internet publishers, or distributors of search and paid search advertising technology.

I’d say dropping Yahoo Site Explorer did harm to both users of internet search and internet publishers. Yahoo Site Explorer was a distinctive tool that only Yahoo offered, allowing both parties named by the DOJ to better understand the inner workings of the search engines they depend on. It also reduced competitive pressure for Google to offer its own tool.

Indeed, things have gotten worse since Yahoo Site Explorer closed. At the end of last December, Bing officially confirmed that it no longer supports the link command in its help forum.

Next To Go, The Link Command?

The link command allows you to enter any page’s web address prefaced by “link:” in order to find links that point at that page. It’s a long-standing command that has worked for many major search engines as far back to late 1995, when AltaVista launched.

Google still supports this command to show some (but not all) of the links it knows about that point at pages. I’d link to Google’s documentation of this, but the company quietly dropped that some time around May 2008. Here’s what it used to say:

Here’s how the command still works at Google. Below, I used it to see what links Google says point to the home page of the official Rick Santorum campaign web site:

The first arrow shows you how the command is being used. The second arrow shows you how Google is reporting there are 111 links pointing to the page. Imagine that. Rick Santorum, currently a major Republican candidate for US president, and Google says only 111 other pages link to his web site’s home page.

The reality is that many more pages probably link over. Google’s counting them but not showing the total number to people who care about what exactly is being considered. I’ll demonstrate this more in a moment, but look at the worse situation on Bing:

One link. That’s all Bing reports that it knows about to those in the general public who may care to discover how many links are pointing to the Rick Santorum web site.

It’s Not Just An SEO Thing

People do care, believe me. I actually started writing this article last Monday and got interrupted when I had to cover how Google might have been involved with a link buying scheme to help its Chrome browser rank better in Google’s own search results.

I doubted that was really the main intent of the marketing campaign that Google authorized (Google did err on the side of caution and punished itself), but the lack of decent link reporting tools from Google itself left me unable to fully assess this as an independent third-party.

As soon as that story was over, renewed attention was focused on why Rick Santorum’s campaign web site wasn’t outranking a long-standing anti-Santorum web site that defines “santorum” as a by-product of anal sex.

Major media outlets were all over that story. My analysis was cited by The Economist, CNN, The Telegraph, The New York Times, MSNBC and Marketplace, to name only some.

But again, I — or anyone who really cared — was unable to see the full links that Google knew about pointing at both sites, much less the crucial anchor text that people were using to describe those sites. Only Google really knew what Google knew.

Third Party Options Good But Not The Solution

If you haven’t heard more complaints over the closure of Yahoo Site Explorer, and the pullback on link data in general, that’s because there are third-party alternatives such as Majestic Site Explorer or the tool I often use, SEOmoz’s Open Site Explorer.

These tools highlight just how little the search engines themselves show you. Consider this backlink report from Open Site Explorer for the Rick Santorum campaign’s home page:

The first arrow shows how 3,581 links are seen pointing at the page. Remember Google, reporting only 111? Or Bing, reporting only 1?

The next two arrows show the “external” links pointing at both the Santorum home page and the anti-Santorum home page. These are links from outsiders, pointing at each page. You can see that the anti-Santorum page has four times as many links pointing at it than the Santorum campaign page, a clue as to why it does so much better for a search on “santorum.”

But it’s not just number of links. Using other reports, I can see that thousands of links leading to both sites have the text “santorum” in the links themselves, which is why they both are in the top results for that word.

Because the anti-site has so many more links that say “santorum” and “spreading santorum,” that probably helps it outrank the campaign site on the single word. But because the official site has a healthy number from sources including places like the BBC saying “rick santorum” in the links, that — along with its domain name of ricksantorum.com — might help it rank better for “rick santorum.”

It’s nice that I can use a third party tool to perform this type of analysis, but I shouldn’t have to. It’s simply crazy — and wrong — that both Google and Bing send searchers and publishers away from their own search engines to understand this.

For one, the third party tools don’t actually know exactly what the search engines themselves are counting as links. They’re making their own estimates based on their own crawls of the web, but that doesn’t exactly match what Google and Bing  know (though it can be pretty good).

Not Listing Links Is Like Not Listing Ingredients

For another, the search engines should simply be telling people directly what they count. Links are a core part of the “ingredients” used to create the search engine’s results. If someone wants to know if those search results are healthy eating, then the ingredients should be shared.

Yes, Google and Bing will both report link data about a publisher’s own registered site. But it’s time for both of them to let anyone look up link data about any site.

The Blekko search engine does this, allowing anyone logged in to see the backlinks to a listed page. Heck, Blekko will even give you a badge you can place on your page to show off your links, just as Yahoo used to. But for Google, it’s “normal” for its link command to not show all the links to a page. Seriously, that’s what Google’s help page says.

Google, in particular, has made much of wanting people to report spam found in its search results. If it really wants that type of help, then it needs to ensure SEOs have better tools to diagnose the spam. That means providing link data for any URL, along with anchor text reporting.

Google has also made much about the need for companies to be open, in particular pushing for the idea that social connection should be visible. Google has wanted that, because until Google+ was launched, Google had a tough time seeing the type of social connections that Facebook knew about.

Links are effectively the social connections that Google measures between pages. If social connections should be shared with the world, then Google should be sharing link connections too, rather than coming off as hypocritical.

Finally, it doesn’t matter if only a tiny number of Google or Bing users want to do this type of link analysis. That’s often the pushback when this issue comes up, that so few do these type of requests.

Relatively few people might read the ingredients labels on the food they eat. But for the few that do, or for anyone who suddenly decides they want to know more, that label should be provided. So, too, should Google and Bing provide link data about any site.

Goodbye Keyword Referrer Data

While I’m concerned about the pullback on link data, I’m more concerned about how last October, Google stopped reporting to publishers the keywords people used to find their web sites, for times when those people were logged into Google.

Link data has long been suppressed by Google. Holding back on keyword data is a new encroachment.

Google has said this was done to protect user privacy. I have no doubt many in the company honestly believe this. But it if was really meant to protect privacy, then Google shouldn’t have deliberately left open a giant hole that continues to provide this data to its paid advertisers.

Worse, if Google were really serious about protecting the privacy of search terms, then it would disable the passing of referrers in its Chrome browser. That hasn’t happened.

Unlike the long examination of link data above, I’ll be far more brief about the situation with Google withholding link data. That’s because I’ve already written over 3,000 words looking at the situation in depth last October, and that still holds up. So please see my previous article, Google Puts A Price On Privacy, to understand more.

Google’s Weak Defense

Since my October story, the best defense that Google’s been able to concoct for withholding keyword data from non-advertisers is a convoluted, far-fetched argument that makes its case worse, not better.

Google says that potentially, advertisers might buy ads for so many different keywords that even if referrer data was also blocked for them, the advertisers could still learn what terms were searched for by looking through their AdWords campaign records.

For example, let’s say someone did a search on Google for “Travenor Johannisoon income tax evasion settlement.” I’ve made this up. As I write this, there are no web pages matching a Google search for “Travenor Johannisoon” at all. But…

• If this were a real person, and
• someone did that search, and
• if a page appeared in Google’s results, and
• someone clicked on that page…

then the search terms would be passed along to the web site hosting the page.

Potentially, this could reveal to a publisher looking at their web analytics that there might be a settlement for income tax evasion for involving a “Travenor Johannisoon.” If the publisher starting poking around, perhaps they might uncover this type of information.

Of course, it could be that there is no such settlement at all. Maybe it’s just a rumor. Anyone can search for anything which doesn’t make it into a fact.

More likely, the search terms are so buried in all the web analytics data that the site normally receives that this particular search isn’t noticed at all, much less investigated.

Extra Safe Isn’t Extra Safe

Still, to be extra safe, Google has stopped passing along keyword data when people are signed-in. Stopped, except to its advertisers. Like I said, Google argues that potentially advertisers might still see this information even if they were also blocked.

For instance, say someone runs an ad matching any searches with “income tax evasion” in them. If someone clicked on the ad after doing a search for “Travenor Johannisoon income tax evasion settlement,” those terms would be passed along though the AdWords system to the advertiser, even though the referrer might pass nothing to the advertiser’s web analytics system.

So, why bother blocking?

Yes, this could happen. But if the point is to make things more private, then blocking referrers for both advertisers and non-advertisers would still make things harder. Indeed, Google still has other “holes” where “Travenor Johannisoon” might find his privacy exposed just as happens potentially with AdWords.

For example, if someone did enough searches on the topic of Travenor and tax evasion, that might cause it to appear one of Google Instant’s suggested searches.

So why bother blocking?

Also, while Google blocks search terms from logged-in users in referrer data, those same searches are not blocked from the keyword data it reports to publishers through Google Webmaster Central. That means the Travenor search terms could show up there.

So why bother blocking?

Nothing has changed my view that, despite Google’s good intentions, its policy of blocking referrers only for non-advertisers is incredibly hypocritical. Google purports this is done to protect privacy, but it puts its own needs and advertisers desires above privacy.

Blocking referrers is a completely separate issue from encrypting the search results themselves. That’s good and should be continued. But Google is deliberately breaking how such encryption works to pass along referrer data to its advertisers. Instead, Google should block them for everyone or block them for no one. Don’t play favorites with your advertisers.

What Google & Bing Should Do

Made it this far? Then here’s the recap and action items for moving forward.

Bing should restore its link command, if not create a new Bing Site Explorer. Google should make sure that its link command reports links fully and consider its own version of a Google Site Explorer. With both, the ability for anchor text reports about any site is a must.

If there are concerns about scraping or server load, make these tools you can only use when logged in. But Yahoo managed to provide such a tool. Blekko is providing such statistics. Tiny third-party companies are doing it. The major search engines can handle it.

As for the referrer data, Google needs to immediately expand the amount of data that Google Webmaster Central reports. Currently, up to 10,000 terms (Google says up to 1,000, but we believe that’s wrong) for the past 30 days are shown.

In November, the head of Google’s spam team Matt Cutts — who’s also been involved with the encryption process — said at the Pubcon conference that Google is considering expanding the time period to 60 days or the queries to 2,000 (as said, we think — heck, we can see, they already provide more than this). Slightly more people wanted more time than more keywords shown.

I think Google should do more than 60 days. I think it should be providing continuous reporting and holding that data historically on behalf of sites, if it’s going to block referrers. Google is already destroying historical benchmarks that publishers have maintained. Google’s already allowed data to be lost for those publishers, because they didn’t begin to go in each day and download the latest information.

So far, all Google’s done is provide an Python script to make downloading easier. That’s not enough. Google should provide historical data, covering a big chunk of the terms that a site receives. It’s the right thing to do, and it should have been done already.

What Publishers Can Do

An anti-SOPA-like effort as targeted GoDaddy isn’t going to work with the search engines. That’s because the two biggest things that publishers could “transfer” out of Google and Bing are their ads and their web sites. But there’s no place to transfer these to that wouldn’t hurt the publishers with incredible amounts of lost traffic.

This doesn’t mean that publishers are powerless, however.

Bing is desperate to be seen as the “good” search engine against “evil” Google. Publishers should, whenever relevant, remind Bing that it’s pretty evil not to have maintained its own version of Yahoo Site Explorer much less to have closed the link command.

Mention it in blog posts. Mention it in tweets. Bring it up at conferences. Don’t let it die. Ask Bing why it can’t do what little Blekko can.

As for Google, pressure over link data is probably best expressed in terms of relevancy. Why is Google deliberately preventing this type of information from being studied? Is it more afraid that doing so will reveal weaknesses in its relevancy, rather than potential spam issues? Change the debate to relevancy, and that gets Google’s attention — plus the attention of non-publishers.

There’s also the issue of openness. Google shouldn’t be allowed to preach being “open” selectively, staying closed when it suits Google, without some really good arguments for remaining closed. On withholding link data, those “closed” arguments no longer stand up.

As for the referrer data, Google should be challenged in three ways.

First, the FTC will be talking to publishers as part of its anti-trust investgation into Google’s business practices. Publishers, if asked, should note that by withholding referrer data except for Google’s advertisers, it’s potentially harming competing retargeting services that publishers might prefer to use. Anti-trust allegations seem to really get Google’s attention, so make that wheel squeak.

Second, question why Google is deliberately leaving a privacy hole open for the searchers it’s supposedly trying to protect. If Google’s really worried about what search terms reveal, the company needs a systematic way to scrub potentially revealing queries from everything: suggested searches, reporting in Google Webmaster Central, AdWords reporting as well as referrer data.

Finally, withhold your own data. Are you opted-in to the data sharing on Google Analytics that launched back in 2008? Consider opting-out, if so:

To opt-out, when you log in, select an account, then select “Edit Analytics Account” next to the name of the account in the Overview window, then you’ll see options as shown above and as explained on this help page.

Opting out means you can’t use the benchmarking feature (fair enough, and no loss if you don’t use it) and Conversion Optimizer. If you still want Conversion Optimizer, don’t opt-out or alternatively, tell Google that you should have a choice to share data solely for use with that product but not other Google products.

There might be other drawbacks to not sharing that I’m missing. But we haven’t been sharing here at Search Engine Land since the beginning of the year. So far, we’re not having any problems.

Google loves data. Withholding your own is another way for publishers to register their displeasure about having data withheld from them. And it’s the type of thing that Google just might notice.

Related Articles

• What Is SEO / Search Engine Optimization?
• Search Engine Land’s Guide To SEO
• The Periodic Table Of SEO Ranking Factors
• What Is Google PageRank? A Guide For Searchers & Webmasters
• How The “Focus On First” Helps Hide Google’s Relevancy Problems
• Yahoo Completes Global Organic Transition To Bing
• Yahoo Site Explorer Closing Down Monday, November 21st
• The Microsoft-Yahoo Search Deal, In Simple Terms
• Google Releases New Link Reporting Tools
• Google Now Reporting Anchor Text Phrases
• Yahoo Adds Link Badge In Site Explorer
• Link Building Tool Review: Link Research Tool Set By CEMPER
• Link Building Tool Review: SEOmoz PRO
• Link Building Tool Review: SEO Book
• Link Building Tool Review: Raven Tools
• Link Building Tool Review: Majestic SEO
• Bing Webmaster Tools Launches New Link Reports; Google Webmaster Tools Changes Theirs
• Blekko’s SEO Tools: What Information Do They Provide?
• Blekko Offers New Linkroll Widget & More Publisher Tools
• Google’s Chrome Page No Longer Ranks For “Browser” After Sponsored Post Penalty
• Should Rick Santorum’s “Google Problem” Be Fixed?
• How Rick Santorum Is Making His “Google Problem” Worse
• For “Define An English Person,” Google Suggests The C-Word
• Google’s Spam Report Page Gets “Biggest Refresh” In Years
• Google: As Open As It Wants To Be
• Google’s Facebook Competitor, The Google+ Social Network, Finally Arrives
• How Google Instant’s Autocomplete Suggestions Work
• Google Webmaster Tools Adds Page-Level Query Data
• Google Webmaster Tools Search Queries Report Now Less Accurate
• Google Analytics Benchmarking Feature, Data Sharing & Audio Ad Charting
• Guest Opinion: Is Google’s Privacy Move Really An Anti-Competitive Practice?
• Dear Congress: It’s Not OK Not To Know How Search Engines Work, Either
• Googleopoly: The Definitive Guide To Antitrust Investigations Against Google
• Keyword “Not Provided” By Google Spikes, Now 7-14% In Cases
• Google Puts A Price On Privacy

The short Yahoo blog post reads:

With the completion of algorithmic transition to Bing, Yahoo! Search has merged Site Explorer into Bing Webmaster Tools. Webmasters should now be using the Bing Webmaster Tools to ensure that their websites continue to get high quality organic search traffic from Bing and Yahoo!. Site Explorer services will not be available from November 21, 2011.

Bing Webmaster Tools – as Google Webmaster Central does — will provide linkage data about a site that has been claimed within their systems. But neither provides information about sites beyond those that a publisher owns (see Bing Webmaster Tools Launches New Link Reports; Google Webmaster Tools Changes Theirs for more about these tools).

For years, Google has deliberately suppressed reporting all the links. This means with the loss of Yahoo Site Explorer, there is no tool from any major search engine that will show all the links pointing at any site on the web.

Small search engine Blekko, however, does provide these types of tools. And, there are third party free tools such as Majestic Site Explorer,  and SEOmoz Open Site Explorer.

Postscript From Danny Sullivan: Now Yahoo Site Explorer has officially closed, with this page left:

Yahoo leaves some related links to its other services at the bottom, most of which make some sense in relation to the original audience of Yahoo Site Explorer. However, “Web Hosting” — which the arrow points to — kind of stands out.

Yahoo offers web hosting as part of its Yahoo Small Business area, which also offers things like domain registration and online stores. Why not point to the entire small business area, then?

My guess is that someone had the bright idea that all those links pointing to Yahoo Site Explorer over the years should be used to channel some potential traffic and ranking boost to the web hosting area. It’s not even identified as “Yahoo Web Hosting.”

Related Stories

• Yahoo Shutting Down Site Explorer This Year
• Bing Webmaster Tools Launches New Link Reports; Google Webmaster Tools Changes Theirs
• Blekko’s SEO Tools: What Information Do They Provide?
• Blekko Offers New Linkroll Widget & More Publisher Tools
•  Link Building Tool Review: Link Research Tool Set By CEMPER
• Link Building Tool Review: SEOmoz PRO
• Link Building Tool Review: SEO Book
• Link Building Tool Review: Raven Tools
• How To Terrify Executives Into Linkbuilding
• Worthy Alternatives To The Useless SEO Data Provided By Search Engines

Image credit: kaarsten / ShutterStock.

This isn’t unexpected, and I am personally surprised it has been live for so long. That being said, the tool that launched on September 29, 2005 will be closing down “later this year” according to Hemant Minocha, Product Manager, Yahoo Site Explorer.

Yahoo recommends that webmasters sign up with Bing Webmaster Tools and use that as Yahoo shuts off Site Explorer.

Yahoo makes it sound like that you won’t miss the old Site Explorer but I find it hard to believe that Microsoft will provide detailed link data for competitors. Of course there are plenty of third party tools to get at this data, such as Majestic SEO and SEOmoz but with the loss of Site Explorer, there will likely be no search engine-provided look at your back links. Yahoo said:

The Bing Webmaster Center team has been actively adding new features over the past few months, including detailed traffic statistics and a new inbound links feature. Microsoft is also committed to build analytic reports that include Yahoo! search usage.

For more details, see the Yahoo Search Blog. No specific closure date has been set, but the APIs are set to close on Sept. 15, so it seems likely the service itself will close on that date.

Related Articles:

• Bing, Yahoo Discussing Future Of Yahoo Site Explorer
• Yahoo Site Explorer: What’s The Status Of the Link Data?
• SEOmoz Launches Open Site Explorer, A Competitor (Replacement?) To Yahoo Site Explorer
• Yahoo-Microsoft Organic Transition Happening, Site Explorer, Search Monkey Holding For Now

• Google Webmaster Tools has added “total links” to the revamped link reports.
• Yahoo seems to have fixed the broken site explorer linkage data.
• Google has fixed the blogspot indexing bug and now can index new blogspot blog posts.
• Google Webmaster Tool’s fetch as Googlebot is 404ing for some webmasters
• Google sometimes is using your heading tags for snippets as opposed to your title tags
• SEOs are noticing major ranking changes since Thursday of last week.

I believe that recaps some of the more recent SEO discussions from over the weekend that deserve your attention.

We are working with the Microsoft team to ensure that webmasters can continue to use Yahoo! Site Explorer to view inlinks (a.k.a backlinks) to a given webpage or a website and to further be able to export them as CSV if desired. Please note that this functionality will be available only through the Site Explorer UI and not through the Webservices API.

This message aligns with a Yahoo blog post on August 17th that indicated:

The Yahoo! Site Explorer team is planning tighter integration between Site Explorer and Bing Webmaster Center to make the transition as smooth as possible for webmasters…. To keep things simple, we will share site information you provide on Site Explorer with Microsoft during this transition period. When Microsoft fully powers the Yahoo! Search back-end globally, expected in 2012, it will be important for webmasters to use Bing Webmaster Center as well. The Bing tool will manage site, webpage and feed submissions. Yahoo! Site Explorer will shift to focus on new features for webmasters that provide richer analysis of the organic search traffic you get from the Yahoo! network and our partner sites.

But according to my conversion with Yahoo, the data that’s missing now is unrelated to the Yahoo/Bing integration. Yahoo told me, “Yahoo! is investigating this report and will work to resolve it quickly if an issue is found, however it is unrelated to the longer term question we previously addressed regarding future support for backlinks.” Yahoo reiterated that the long term plan is to continue to provide external link data via Site Explorer.

Where Is the Link Data Coming From?

Since Bing is powering Yahoo’s index, is the external link data provided by Site Explorer stale? “Not at all”, says Yahoo. “Link data in Site Explorer is coming from Yahoo’s data today. We continue to maintain Yahoo!’s crawlers and indexers during this transitional period, ensuring fresh data in Yahoo! Site Explorer.  The US and Canada are currently the only markets for which certain back-end functions of Yahoo! Search are being powered by the Microsoft search platform.” However, it does sound like Yahoo will transition to Bing data at some point: “We are working with Microsoft to ensure that we will continue to have comprehensive, fresh link data for Yahoo! Site Explorer when we switch from Yahoo!’s data to Microsoft data in the future.”

Should Site Owners Continue To Provide Data to Site Explorer?

Site Explorer does more than provide external link data. It also enables site owners to provide data to Yahoo. You can submit XML Sitemaps and provide details on how to handle URL parameters, remove URLs from the index. You can also view query data, crawling and indexing stats, crawl errors, and Delicious activity. For now, Yahoo is stressing that (until 2012) all non-US and Canada searches are still powered by the Yahoo index (except, of course, Yahoo Japan, which is now powered by Google), and so if you care about international Yahoo search traffic, you should continue to provide input data (such as XML Sitemaps and URL parameters) to Yahoo. The Delicious data should continue to be accurate, of course. The robots-nocontent tag is likely also only valuable for Yahoo’s international properties, as Microsoft previously decided against supporting it. Does the URL removal functionality still work to remove results from the Bing-powered, Yahoo-displayed index? Are the query stats still accurate for Yahoo searches? Yahoo hasn’t yet commented on these questions or on the value of the indexing stats and key terms (are these stats for Yahoo’s international indices?). And what about crawl errors? The sites I own show “no crawl errors”. Is this because they don’t have any errors or because Yahoo is no longer reporting them? I’ll update as I learn more.

Yahoo Slurp Continues to Crawl the Web

If you review your server logs regularly, you likely have already noticed that Yahoo Slurp continues to crawl the web.  It sounds like this will continue, both for the purposes of Yahoo’s international search properties and for Site Explorer data until early 2012.

Where Else Is External Link Data Available?

Google webmaster tools provides link data to verified site owners (and recently revamped those reports). When Bing reintroduced their webmaster tools, they removed link data, but at SMX East a few weeks ago, they announced they would be providing new link data, although this has yet to launch. For data on competitive links, several data sources exist, including:

• SEOmoz Linkscape
• Majestic SEO
• Exalead link: operator
• SearchDNA
• Link Diagnosis

Yahoo announced that later this week the organic transition will commence for PC and mobile results in North America:

This is an important step toward our goal of improving the overall relevance of Yahoo! organic search results and attracting a larger audience to Yahoo! Search, to ultimately put your ads in front of more potential customers. You’ll want to make sure that you’re prepared for this change, so be sure to check out these tips and stay tuned to the Yahoo! Search blog for confirmation of when the organic search transition is complete.

A “powered by Bing” message at the bottom of results will indicate that these results are live.

Yahoo also said that limited testing of paid account transitions has begun. There will be a “transition” portal with “simple step-by-step [instructions for] creating a Microsoft Advertising adCenter account and importing your campaigns, or linking an existing adCenter account you may already have.”

Search Monkey will continue on but with substantial changes:

All of the existing enhanced result templates will continue to be generated from websites’ page markup and structured data feeds, and Yahoo! will continue to show this structured data on the Yahoo! Search results page, along with Microsoft’s organic listings. Over time, some of this structured data processing will be supported natively by the Microsoft platform. Webmasters will continue to have the ability to affect the presentation of a search result through page markup on their site (microformats and RDFa).

As we look to the future of Yahoo! Search, we are focusing on new search-related offerings we believe will provide additional value for publishers and partners.  In order to align our resources on strategic priorities, we have decided to close the SearchMonkey developer tool, gallery, and app preferences on October 1, 2010.  As a result, third party custom result apps, infobar apps, and data services will no longer appear on Yahoo!’s search results.  For developers who wish to retain their code, please export it using your favorite copy/paste tool before then.

Site Explorer will continue to be used by Yahoo until 2012 when the organic transition is complete:

[Y]ou should continue to provide your site information to Yahoo! using Site Explorer, because in many parts of the world search results will continue to be powered by Yahoo!’s systems until the full transition is complete for all markets by early 2012. This will help to ensure that you continue to get high quality traffic from searches originating on Yahoo! and our partner sites, even in markets that are not yet transitioned to Microsoft’s systems. We will share site information that you provide on Site Explorer with Microsoft during this transition period, to ensure that you get high quality traffic from search results that are powered by Bing also.

However Yahoo also recommends becoming familiar with the Bing Webmaster Center tool in the interim.

Related posts:

• All New Microsoft Bing Webmaster Tools
• What Site Owners, Web Developers & SEOs Should Know About The Yahoo Microsoft Deal

Postscript by Barry Schwartz: I spoke with Yahoo’s Shashi Seth after this announcement and I wanted to clarify that Site Explorer will ultimately be fully integrated into Bing Webmaster Center. Currently, the new Bing webmaster tools is missing a link tool completely. I suspect that was planned and Yahoo’s Site Explorer features will be that link tool in Bing Webmaster Center. We should know more about when Site Explorer’s integration schedule within the next 30 days or so.

Yahoo’s Arnab Bhattacharjee told conference attendees that the two companies are now going through discussions on how to transition from being competitors to the new situation where Bing will power Yahoo’s algorithmic search results, and that Site Explorer is part of those discussions.

Sasi Parthasarathy of Bing perhaps offered some reassurance when he said that users come first — what is most important to users will determine what products stay.

Parthasarathy said they hope to have an announcement soon.

Open Site Explorer provides a number of data points pulled from SEOmoz’s Linkscape tool and its index of the web. It shows stats such as overall link counts and a count of domains that link to a URL, along with anchor text distribution and more. A helpful feature is the ability to compare two domains side by side.

Although it’s called “open,” the tool is limited for those who aren’t SEOmoz Pro members. Guests are allowed three reports per day, and those reports will show only up to 200 links per report. SEOmoz free account holders get unlimited reports and can see up to 1,000 links per report — this is the level that seems to be most similar to what Yahoo Site Explorer offers. Paying SEOmoz Pro members get unlimited reports, up to 10,000 links per report, and a variety of other metrics not available to anyone else.

Is it a competitor for Yahoo’s Site Explorer? Most certainly. Is it a possible replacement? Well, that depends on what happens if and when the Microsoft-Yahoo search deal gets approved. When asked last night about the future of Yahoo Site Explorer, spokespersons from both companies said they couldn’t comment.

• Key Terms
• Delicious Activities
• Top Delicious Tags
• SearchMonkey Objects

Barry posted this screenshot showing the new features as they related to Search Engine Roundtable:

It was just a month ago that Yahoo expanded Site Explorer with a new “Top Queries/Top URLs” feature. Webmasters and search marketers still wonder, though, what Site Explorer’s fate will be if the Yahoo/Microsoft search deal gets approved. Yahoo hasn’t commented on our questions about that.

Postscript by Barry Schwartz: Yahoo has now announced it on the Yahoo Search Blog.

The last time we reported about new Site Explorer features was over a year ago. Back then, the action bar had less features, the one on the left is the old one and the one on the right is the new one:

Note, I did crop the bottom of the new one off, simply because that matches the old one.

Here is a closer look at these reports: