According to the Wall Street Journal, Cutts is in South Korea this week and, in a presentation Monday night for about 80 government officials, webmasters, lawyers and journalists, managed to singlehandedly convince some government reps to let Googlebot crawl and index their websites.

One of those in the audience was Kang Min-koo, a senior judge in the Seoul High Court. When he saw the court’s Web site was on Mr. Cutts’ list of government sites that couldn’t be indexed by Google – and thus couldn’t be found on a Google search – he sent a text message by phone to the court’s webmaster ordering it to be changed.

Since the change can be made by altering just a few lines of software code, the webmaster had it done in no time. When it came time for questions, Mr. Kang asked Mr. Cutts to check if the High Court’s site showed up on Google – and it did.

Cutts’ visit to South Korea comes on the heels of tension between the government and Google. Earlier this month, the Korean Fair Trade Commission accused Google of interfering with its Android antitrust investigation. The commission alleges that Google deleted documents pertinent to its investigation into whether Google is limiting access to local search engines on Android smartphones.

There’s also the matter that South Korea is one of only a handful of countries where Google isn’t the dominant search engine. And, as the WSJ points out, Google isn’t likely to gain market share in Korea if prominent websites aren’t in its index.

While Cutts may have the title of being Google’s chief spam cop, he’s long been one of the companies go-to public faces. A little more than a year ago, Google sent him to Washington, DC on an “educational tour” aimed at telling government officials that Google’s search results don’t need to be regulated.

Most Google users are happy, Singhal said. Of course, Search Plus Your World isn’t perfect, he admits, but it’ll improve. As for including content from social networks like Twitter and Facebook, Google’s open to that, as long as long-term deals can be reached that give Google faith it can build new features that will last.

I talked today with Singhal — the “Google Fellow” who oversees all of Google’s search algorithms — about the reaction to Search Plus Your World. Here’s the Q&A, questions in bold, answers from Singhal indented below them.

What’s been your overall reaction or thoughts as you’ve seen all the debate about Search Plus Your World?

The overall takeaway that I have in my mind is that people are judging a product and an overall direction that we have in the first two weeks of a launch, where we are producing a product for the long term.

We’re clearly not done. The product is not complete. It will improve, and we are going to add more things to it. We — Google — always launch products and learn from our launches.

Let me also add one thing. Here is something that I’ve been noticing. The user feedback we have been getting has been almost the other side of the reaction we’ve seen in the blogosphere. The users who have seen this in the wild are liking it, and our initial data analysis is showing the same.

With Universal Search or Google Instant, there were more initial complaints, then even later, people were delighted with it.

NOTE: Earlier this week, I was reviewing Google’s web search support forums myself. There really isn’t any massive outcry as we’ve seen with other launches, such as when Google Reader integrated Google+ sharing last year.

There are two main changes that Search Plus Your World introduced. One was the ability to search through privately-shared content along with public content. Some have written they dislike this. Your reaction?

Every time a real user is getting those results, they really are delighted. Given how personal this product is, you can only judge it based on personal experiences or by aggregate numbers you can observe through click-through.

Like everything else, we will improve the rankings and so on. But out of the gate, whereas we had limited users to train this system with, I’m actually very happy with the outcome of the personal results.

How about the concerns that Google+ is being favored too much?

Once again, the blogosphere, they’re just judging the book by the cover.

NOTE: I then mentioned there have been some examples of these blogged, such as by John Battelle, Mat Honan and just today, Tom Blue wrote about how personalized results made searches for things like “television” or “knives” seem to have irrelevant listings. Singhal then said….

I wouldn’t disagree that there may be a few examples out there where our system could be better, but there are millions of queries where our system could be better.

But specifically, there are concerns Google+ is being favored too much not in the privately-shared results but rather as suggestions in the search box or in the new People & Pages results.

This goes back to the point we have been making, that we only have access to personal content and personally shared content from Google Plus. We don’t have that access from other companies.

But what about the Don’t Be Evil tool launched this week, showing that you can create People & Pages results that do go beyond Google+ content.

This is where I’m saying most people are jumping to a conclusion based on the first two weeks of the product. We’re designing a product which it will work for all individuals. It will have identities as a fundamental ingredient of search and relationships as another fundamental ingredient of search.

All this debate is centered around these very popular people out there, and what you could have done for them, and what makes the most sense. But when you’re developing a product, you don’t want to develop it for one segment of the population.

I’ve not seen the debate that I have a [personal] friend named Ben Smith, and when I try to search for him, I get bad results. If you actually build a product only for popular people, then it undermines what we are trying to do. You can’t build a product that behaves differently for one class of people then differently for the real relationships that matter for you.

I feel like this point has not been emphasized enough. People have gotten stuck on these very popular names, and no one has been focused on the cool part of the product where you can find people you care about.

But the People & Pages section does seem focused on famous people or big companies right now.

This is the first two weeks of the product. We have designed for the future of the product. We have designed a product not just for how it works for today, and what we really want is for it to work for tomorrow. Going forward, we’d be interested in doing things like if you’re looking for a destination, we’d like to present to you people that you know who may have something to say about that destination.

People are coming to a conclusion about the product today, within the first two weeks, and they’re not fully seeing the potential where we can build this product around real identities and real relationships.

Is what we have now in People & Pages a promotion for Google+ or search results meant to be more than pushing Google+?

What we are thinking about that product is that if you’re interested in something, who are the people you are potentially related to that you should know about? If they have shared things with you, that’s the obvious thing. It’s easier to build the early system with popular people, but that’s just the beginning of the product.

But how about now? Is this more a Google+ promotional unit?

We don’t think of this as a promotional unit now. This is a place that you would find people with real identities who would be interesting for your queries.

Currently, that place I would admit is occupied by really popular people. So what we are trying to do with this product is indeed bring that aspect that there’s this real society out there that you should know about.

Why not show the additional social links there? For example, you show Britney Spears with her Google+ page in a search for “music.” She links to her Twitter and Facebook accounts from her Google+ page. Couldn’t those be listed as alternative links within People & Pages?

If people click on her Google+ page, those links are available on her About page. Once again, this is possible for us [to maybe show alternative links] because we know much more about the real identities on Google Plus. We can’t provide that for other [lesser known] people.

But what if the people I really want to know about are on those other social networks?

Now I go back to watching how people are acting [with Search Plus Your World]. For example, if someone is not active on Google+, indeed, we shouldn’t be showing their very old posts. That’s one big lesson I’ve learned from watching this in the wild. We’re learning how we can improve the product. We’re already starting to launch some of these improvements as we speak.

What do you need from companies like Twitter and Facebook to integrate them into Search Plus Your World?

Fundamentally, what we learned with our great Realtime Search product is that once you build a great product that users love, then someone else can decide the fate of that product [because when the Twitter deal wasn't renewed, the Google's Realtime Search service depended so much on Twitter that it had to be closed].

That was a very bad experience for Google’s users, and it was a bad experience for our teams. They put their heart and souls into building a great product, just to see that go to waste.

We’re very open to incorporating information from other services, but that needs to be done on terms that wouldn’t change in a short period of time and make our products vanish.

What if you got the Twitter firehose of data without needing a deal. Would that work?

The question that comes down to is under what terms all that’s under. I’m just very wary of building a product where the terms can be changed.

But what if you were given all the posts that people make, no deal needed, just full access to the firehose. Would that be enough?

There’s more to it. A good product can only be built where we understand who’s who and who is related to whom. Relationships are also important alongside content. To build a good product, we have to do all types of processing. But fundamentally, it’s not just about content. It’s about identity, relationships and content. Anything else trivializes a very hard product.

What about if you were also given a feed about identities and relationships. Who someone is, all their social connections that are made public?

We’re very open to talking to all the parties on what does it take to build a proper identity product for users, with proper relationships managed.

Wasn’t something like the Open Social Graph API supposed to provide this? Or aren’t there these type of standards already?

I wish there were such standards that were widely adopted out there. One problem we already have observed is there are all types of link spam connections on Google Plus. We have to develop sophisticated analysis, say if someone posts with a particular pattern, we can tell they aren’t a real person.

There’s a lot more. It’s not just about content. It’s about identity, and when you start talking about these things and what it takes to build this, the data needed is much more than we can publicly crawl.

Moving Forward

That’s the interview. On a personal note, I’ve viewed the debate around Search Plus Your World as perhaps an opportunity to break the stalemate that’s existed between Facebook and Google for years over sharing data, as well as resolve the loss of Twitter data that happened last year.

Both Facebook and Twitter have real reasons to fear that Google — with its own Google+ social network — might use their data in a way that would threaten their own businesses.

However, both Facebook and Twitter also have good reasons for wanting to be better included in Google’s search engine. Similarly, Google’s search engine would benefit by having more social content within it, as well as a better understanding of social relationships.

That also means I think Google’s search engine would benefit if it could socially connect with any social network, not just Google Plus. I especially want to see the Google search engine continuing to do what I’ve depended on it to do, list the best content from across the entire web, regardless of where it is located.

I covered the strange balancing act Google has to walk between its social network and its search engine at the end of my article about the Don’t Be Evil tool from Facebook and Twitter that launched this week.

As for the balancing act Facebook and Twitter face with Google, I covered more about that — and some potential solutions — in my other post from earlier this this week: A Proposal For Social Network Détente.

Please consider reading both of those articles to understand more about these issues. There’s also related background reading below.

Related: Twitter-Google Deal

• Google Launches Real Time Search Results
• As Deal With Twitter Expires, Google Realtime Search Goes Offline
• Google Realtime Search & The Aftermath Of The Google-Twitter Split
• Twitter Renews Deal With Bing; Google Deal Remains MIA

Related: Facebook-Google Data Dispute

• Facebook On Social Search: ‘We Want To Work With Everybody’
• Facebook: You’ve No Right To Export Email Addresses (Unless It’s To Yahoo & Microsoft)
• Google & Facebook: If You’re So Smart, Work It Out!
• How Facebook Enables The Google Social “Scraping” It’s Upset About
• Google To Close Social Graph API, Not OpenSocial

Related: Google’s Search Plus Your World

• Google’s Results Get More Personal With “Search Plus Your World”
• Schmidt: Google+ Not Favored, Happy To Talk Twitter & Facebook Integration
• Twitter Cries Foul Over Google “@WWE” Search, But Google Still Beats Bing
• “Don’t Be Evil” Tool — Backed By Facebook & Twitter — Shows Google’s “Search Plus Your World” Can Go Beyond Google+
• FAQ: What’s The Debate About Google’s Search Plus Your World?
• A Proposal For Social Network Détente

As The Guardian reports, the groups involved in the proposal include the British Phonographic Industry (BPI), the Motion Picture Association (MPA), the Premier League, the Publishers Association and a TV/film trade group called the Pact. The groups’ suggestions are spelled out in a nine-page document that became public after a Freedom of Information Act request.

The Piracy Problem

The entertainment groups say it is “trivially easy for UK consumers to find and access illegal entertainment content via search engines.”

On September 26 2011, BPI made test searches on Google for the name of each of the UK’s top 20 singles and albums, followed in each case by the word “mp3″ (the dominant legal and illegal file format for digital music). On average, 16 of the first 20 Google results for charts singles and 15 of the top 20 search results for chart albums linked to known illegal sites.

The report also cites a Publishers Association study that involved searches on both Google and Bing for the 50 bestselling books during the last week of April 2011. In that study, “Google returned an average of 41% non-legal links in the top ten (first page) results” and “Bing returned an average of 21% non-legal links in the top ten (first page) results.”

Search Engines Code of Practice

The report suggests a Code of Practice for search engines that includes the following:

• assign lower rankings to sites that “repeatedly make available unlicensed content in breach of copyright”
• prioritize (i.e., give better rankings/visibility to) websites that “obtain certification as a licensed site” for content downloading
• stop indexing websites that are subject to court orders and create “procedures to de-index substantially infringing websites”
• improve the “notice and takedown” system and ensure that searchers aren’t sent to illegal sites via suggested searches, related searches and suggested sites
• ensure that search engines don’t advertise illegal sites, place advertising on them, or profit from selling keywords related to illegal sites
• ensure that they don’t profit from selling mobile apps that “encourage infringement”

Google declined to comment to the Guardian.

The company faced similar pressure in December when the Recording Industry Association of America (RIAA) scolded Google and suggested that the company’s search algorithm be changed to better fight online piracy. Even Congress has discussed whether Google should favor legal sites in its search results.

Google did extend an olive branch of sorts in the U.S. when it launched the Google Music store in November. Earlier last year, Google also began removing some piracy-related terms from its Instant Search product.

The ads argue that Google unfairly promotes its own products, delivers search results that aren’t objective or in the best interests of users and doesn’t respect user privacy. The ads assert that they’re telling the public things that “Google doesn’t want you to know.”

The new members of FairSearch.org include the Interactive Travel Services Association (ITSA), Twenga, which just filed an antitrust complaint against Google in Europe and ShopCity.com, a local-community shopping destination.

Google has had a run of negative press lately, including alleged bad behavior in Kenya, controversy over Search Plus Your World, its more recent privacy policy changes and the return of the Google “illegal pharma ads” incident with new details surrounding the criminal “sting” against Google.

The combination of a more openly competitive (even combative) attitude coming out of Google, recent sweeping changes in search and a spate of negative press have created new dangers for the company’s public image. While there is no immediate threat to its search and advertising leadership, regulators and lawmakers on both sides of the Atlantic are increasingly skeptical and critical of Google, and competitors have mobilized against the company — as reflected in the “Good to Know” campaign.

Yet, for the most part, the debates surrounding Google’s “fairness” are mostly happening among industry insiders and have yet to penetrate to the level of mainstream awareness. FairSearch.org has been trying to change that with a steady stream of anti-Google PR and messaging.

Related Entries

• US Consumer Group Wants EU To Block G-Moto Deal, Calls For Google Breakup
• WSJ: Federal Prisoner Helped U.S. Sting Against Google’s Pharmacy Ads
• Poll: 66% Say They’ll Cancel Google Accounts Over Privacy Changes
• Google Outspends Microsoft In DC Lobbying In 2011
• EU Ready To Impose Tough Privacy Rules On Google, Facebook
• Do The Majority Of Americans Think Google Is Unfair?
• Does The FairSearch White Paper On Google Being Anticompetitive Hold Up?

Beyond Blame, Believing In Cooperation

Let’s set aside blame, because blame isn’t going to move anything forward.

Let’s also assume that all the players can be taken to some degree at their word, that they do indeed want to work together in some ways.

What does everyone want?

The Wants & A Game Changer

Google has wanted its own social graph for some time. By social graph, I mean a way for it to see how people are interconnected, which could potentially improve Google’s search results. Better ways to target ads, new “sticky” content where people spend huge amounts of time are other things it wants, but let’s focus on the search aspects.

Twitter has wanted, well, clearly money along with other non-disclosed things for what is mistakenly assumed to be its most valuable asset, its “firehose” of tweets. What’s really Twitter’s most valuable asset is actually its sharing activity, but I’ll get back to that.

Facebook has wanted, well, I don’t know — maybe to be the best social network out there. Don’t laugh. If Google once had a laser-like focus on being the best search engine out there, Facebook can have the same focus on social. That’s a vision that Facebook probably doesn’t want ruined by Google encroaching on its territory. A vision that, of course, makes Facebook plenty of money by tapping into social actions.

We also now have something new that both Facebook and Twitter say they want — to be better represented in Google’s search results. It’s not as if they haven’t been there. But releasing today’s Don’t Be Evil bookmarklet — which alters the Google’s Search Plus Your World service — was a game changer in these Cold War years between the search and social superpowers.

The Bill Of Social Data Rights

Can everyone get what they want, all succeeding in their own ways, without further banging of virtual shoes in outrage?

I think they can. Or at least, I think there are some ways forward, some common ground that has emerged. I’ll lay these out in what I’ll call a Bill Of Social Data Rights. I’m open to better names, believe me — that name I know has been used for other things than I’m covering. But hopefully you get the point.

1) Public Is Public

If someone posts something public on a social network, that’s public for any search engine to index. It’s not up to Twitter or Facebook or Google+ to decide if Google or Bing get to index it through some special deal. The content of what was written belongs to the person who wrote it. If that person publishes publicly, then search engines can spider what they find.

Posts are, after all, web pages. People who create them can choose to block them from search engines if they want. People can choose to withdraw public posts after they’ve been published and know that, like with any web page, the posts will eventually be dropped from search engines.

Social networks have lots of content. Search engines like Google and Bing could bring them to a halt, if they tried to grab everything without special arrangements. So make the “firehose” arrangements, and make them on behalf of the users, who might actually want to find their own content in the search engines of their choice.

This helps solves, by the way, the concerns that Facebook and Twitter have raised about not being included enough in Google. Much of their content is included. Much more of it could be included, if deal making was set aside in favor of the Public Is Public principle.

2) Profiles Are Meta Data

If someone creates a public profile, certain types of information from that profile should be expressed in an easy, machine-readable format. Any social network should provide the profile’s name, the number of followers, the number of people being followed, as well as a flag to indicate if a profile is somehow verified or trusted. A trusted way to link that profile to other profiles or web sites should be implemented.

Search engines need this type of information, so that they really know who someone is, in order to return search results. That’s true even if the person is using a pseudonym. You still want to know that the pseudonym account that comes up is the right one, not some faker.

Providing meta data abut profiles will help. Search engines won’t have to guess where to scrape for key information such as follower counts, which can be used to tell if an account might be real or not, in the case of celebrities. Cross-linkage can help avoid problems that both Google and the Don’t Be Evil tool have, where Larry Page gets listed with Facebook and Twitter profiles that aren’t really his.

If Facebook and Twitter really want better inclusion in things like “People & Pages” listings at Google, this type of data will help. Similarly, if Bing wants to include Google+ profiles, it could use the information as well.

3) Your Friends Are Yours

If I have friends on one service, I should be able to import those friends in some machine-readable format to another service. That doesn’t mean a list of text names, as Facebook’s tool will kick out for me.

It might mean email addresses, as Facebook will give me for Yahoo or Microsoft but not for Google.

It really means some way that the geniuses at our social networks can concoct, I’m sure, so that if I want to find my Twitter friends at Google+, I can. If I want Path to know a particular Google+ circle of friends I have, I can do that. If Pinterest wants to know my Facebook friends, they are my friends to take — not Facebook’s to decide.

What is there really to fear, by allowing this? Even if I take my 50 friends to a new social network, if that social network is crap, no great victory has been won. Heck, taking the names doesn’t mean any of the 50 will actually follow me over.

4) Social Shares & Social Actions Are Proprietary

The real gem each of the social networks has isn’t our posts, isn’t our profiles, isn’t our friends. It’s knowing what we do, how we interact with our friends, how we interact with content.

That’s where the walls should remain. Facebook and Twitter, for example, have real reasons to fear that handing over streams of data to Google might allow it to better understand how people are acting on their services.

So don’t. Make content public, yes. But find a way to agree that the actions — the number of tweets, the number of likes, whatever — are declared off-limits for use by other search engines. If things like robots.txt and the nofollow attribute can work for search, the social networks can figure out their own mechanisms.

Moving Forward

Does everyone get what they really want with this?

Google already has a social graph that’s building in Google+. It really doesn’t need to be worried about getting social shares or actions from the other services — though if they want to offer this, that’s their option. But giving Google the additional information I’ve outlined will allow it to do a better job of exposing content from these other services in search.

Twitter doesn’t get some big payday for its feed, but it’s not getting that now. What it does gain is people who stop complaining they can’t find their own tweets. It gains the chance that with decent meta data, Google will rethink the Google+ification of its search results and return to the idea that search results can be social using anyone’s network. And Twitter’s content gains more exposure, important for a service that more and more talks about itself as being a content play.

Facebook gets even more exposure within Google than it has now. Opening up keeps pressure on Google, as with Twitter, that there can’t just be some Google+ification with the excuse that Google has no choice, because Facebook won’t do a deal. And Facebook potentially avoids the anti-trust critics that almost certainly will come for it in a few years, just as they’ve come for Google now, claiming that Facebook is staying to closed and using its market dominance to keep others out.

I know these proposals may sound naive. Some of what I describe might already exist. But we do need a way forward. Search and social have been colliding, but we don’t need a collision. We need a collusion, and not in the negative sense but a collusion where the users really are being served best by the services they depend on. That can happen, even supporting healthy competition, but without the social superpowers going to war.

(Stock image via Shutterstock. Used under license.)

Related Stories

Looking to understand more about some of the issues in the data war between Google, Facebook and Twitter? See especially the first two articles:

• Google’s Results Get More Personal With “Search Plus Your World”
• FAQ: What’s The Debate About Google’s Search Plus Your World?
• “Don’t Be Evil” Tool — Backed By Facebook & Twitter — Shows Google’s “Search Plus Your World” Can Go Beyond Google+

The tool — a bookmarklet that works in your browser — changes three parts of Search Plus Your World that currently shows information only from Google Plus. These are:

• People & Pages results
• Google+ Sitelinks
• Google+ Suggestions In Autocomplete

The tool can be found on the Focus On The User web site. It was created, as the site says, by engineers at Facebook, Twitter and MySpace, who in turn consulted with other unnamed social networking companies. The “Don’t Be Evil” name of the tool comes from Google’s philosophy of wanting to focus on what’s best for users, rather than what’s best for Google.

The companies behind the tool feel Google’s hasn’t focused on what’s best for its users with Search Plus Your World. They have a good point. But the tool makes this point better than all the debates that have happened so far around Search Plus Your Word, because it shows what Google could have done to better serve searchers, if it had wanted to.

Below, how to install the tool and how it works, once you’ve done so, along with a revisiting of some of the issues it highlights.

Installing The Don’t Be Evil Bookmarklet

To get the tool, head to the site and look for the “Try a more relevant Google” button:

Clicking the button will make a new window appear, with a “don’t be evil” button that’s visible:

You click and hold on that button, then drag it to the bookmarks bar in your browser (if you don’t see your bookmarks bar, look in your browser’s options to make this visible):

Once it’s installed, you’ll see it on your on your bookmarks bar:

When you search on Google, you push that button to transform your results, as I’ll cover next. By the way, I’ve tested this with both Google Chrome and Firefox. It should also work with Safari. It doesn’t work with Internet Explorer.

Problem: “People & Pages” Powered By Google+

One of the biggest issues I’ve had with Search Plus (my shorthand name for Search Plus Your World) was the new “People and Pages on Google+” results that appear for some searches. These can show up even if you don’t have a Google+ account or are logged out of Google entirely.

These are the first search results that I’ve ever seen on Google that haven’t been inclusive of sources beyond Google, when those sources are available. To understand more about this, and how it’s such a major departure for Google, see my previous post, To Understand Google Favoritism, Think If Google+ Were YouTube.

Aside from favoring Google in a way that I’ve never seen before, I’ve felt that the results simply weren’t providing the best information to searchers at Google, especially given the prime real estate being given to them. My previous story, Real-Life Examples Of How Google’s “Search Plus” Pushes Google+ Over Relevancy, explains more about that.

The reason for the relevancy gap was simple. Google+ still lacks many important or notable people that might be on the other social networks. Indeed, until last week, Lady Gaga wasn’t on Google+ despite being one of the most followed people on Twitter and Facebook. Her absence meant she’d never have a chance at being one of the music recommendations for the People & Places results that Google was showing.

Including other social accounts in People & Pages results would, as I’ve written repeatedly, done what I expect of Google as a search engine, delivered a great search experience. From one of my past posts:

I’d love to see is Google retool the social suggestions that come up, so they aren’t simply Google+ “People & Pages” results injected into Google but more inclusive of other social sites, as well. Because that type of inclusive search product is what Google does well, and what we expect for the company to provide.

Now the engineers at Facebook, Twitter and MySpace have done what I expected Google to do, create a social suggestions search engine. Let’s see some examples.

Solution: “People & Pages” Powered By Google Search

Let’s start with a search for music. On the left, what you see from Google’s Search Plus. On the right, how things change after you click the Don’t Be Evil bookmarklet:

The first set of arrows at the top show how “People and Pages on Google+” is changed instead to “People and Pages from the Social Web.”

The second set of arrows shows how for Britney Spears, rather than her Google+ profile being shown (where she has 1.6 million followers), her Twitter account is displayed (where she has 12.6 million followers). Underneath her Twitter account, links to her MySpace, Facebook and Tumblr accounts are also shown.

The third set of arrows shows how for Snoop Dogg, rather than his Google+ profile being shown (with 1.4 million followers), his Twitter account (with 6.2 million followers) is shown. Underneath that, links to his MySpace, Facebook and Google+ accounts are displayed.

I’ll explain more about how each profile gets selected, but first, let’s do some more examples. Here’s a search for cars:

For this search, the Google+ page for Ferrari (6,500 followers) is changed to Ferrari’s Facebook page (5.6 million followers), with its YouTube page as an alternative link. Toyota’s Google+ page (180,000 followers) is also changed to the Toyota Facebook page (700,000 followers).

One of my favorite examples to show the failing of Google’s current system is a search for Facebook:

On the left, Google’s regular listings suggest Mark Zuckerberg’s Google+ profile as something people should follow. It really is Zuckerberg’s official page, with over 600,000 followers. However, Zuckerberg has never posted to it since creating that profile shortly after Google+ launched.

On the right, the Don’t Be Evil button transforms that listing into something useful, a pointer to Zuckerberg’s Facebook page (his Google+, Twitter and Quora pages are listed as alternatives.

Also notice the arrows. In this case, I was signed-in to Google+, which is why an “Add to circles” button appear next to the Google+ result. However, when the result was transformed to show a Facebook listing, I got an indication showing that I was already following Zuckerberg on Facebook (if I wasn’t, a Like button allowing me to follow him would have appeared).

One more example, this time to show that there’s more intelligence going on that just stripping out Google+ links and swapping them for any other network. Here’s a search for Google:

The first suggestion is for Google’s own page on Google+, which is changed to Google’s page on Twitter. Notice how there’s a “Follow” button next to the Twitter link, as well.

Under that is Larry Page. His listing isn’t changed. It remains pointing to his Google+ page, which makes sense. To the best of my knowledge, that’s the only place he’s active socially. While LinkedIn, Facebook and Twitter accounts are also shown for him, none of those seem to be real accounts for him.

What’s The Most Relevant Social Account To Show? Ask Google!

We could debate which is the most best social account to show for each of the examples above in various ways, such as:

• Should it be based on followers? If so, in many cases, accounts on the established social networks of Twitter and Facebook would easily trump Google+ accounts, though not always.
• Should it be based on activity? Britney hasn’t posted to Google+ since January 18. Her last tweets are more recent, as of January 21. Does that count for more?
• Should it be based on engagement? Even if Britney hasn’t been on Google+ recently, perhaps her engagement with fans is better there?

Rather than try to figure all this out — and potentially come under accusations they were selecting metrics to favor their own social networks — the developers of the Don’t Be Evil bookmarklet decided to let Google itself figure out which social account was most relevant.

For example, let’s go back to how the listing for Snoop Dogg was changed:

What happens is that the bookmarklet looks at which accounts are listed initially by Google in the People & Pages section. It then does a search for the people or companies listed to find which social profiles Google itself believes are most relevant, in the first 100 results.

In the case of Snoop Dogg, the bookmarklet sees that he’s listed as a People & Pages suggestion for a search on “music.” It then conducts a search for “snoop dogg” on Google, to see which social account Google lists first for him.

As you can see above, Twitter is the first social account shown, so the bookmarklet rewrites Snoop Dogg’s suggestion to link to his Twitter account. His MySpace account is found next, so it’s listed as the first alternative. Facebook is found after that, so it’s the second alternative. Google+ is found after that, so it becomes the third alternative.

According to the tool’s FAQ page, it checks for links from any of these social networks:

• CrunchBase (weird, I wouldn’t consider this a social network)
• Github
• Google+
• Facebook
• Flickr
• Foursquare
• FriendFeed
• LinkedIn
• MySpace
• Quora
• Stack Overflow
• Tumblr
• Twitter

All the information the tool finds comes solely from Google itself, even the profile pictures, which are generated using Google’s own rich snippets testing tool, as the FAQ page explains.

Because only Google’s own information is being used, the tool can’t patch up a key weakness, which is that all the suggestions are initially based on people who are in Google Plus.

For example, you will never get Katy Perry as a music suggestion, despite her being one of the most followed musicians on Facebook or Twitter. That’s because she’s not on Google+, so she has no chance of appearing as a suggestion to music-related queries. If she doesn’t get suggested, then the tool can’t add her.

Problem: Autocomplete Profiles Only For Google+

Another feature of Search Plus that favors Google+ is within Google’s search box. As you type, the autocomplete feature of Google Instant will suggest things you should search for. For notable or prominent people, these suggestions may also include a link to their Google+ profile. For example, consider this search for Facebook chief operating officer Sheryl Sandberg:

Just typing in the word “sheryl”  – when I was completely logged out of Google — was enough for it to suggest that I select her Google+ page, where Sandberg last posted in September. That’s pretty preferential treatment. Why should that be suggested over Sandberg’s personal web site or another social profile?

Solution: Expand The Choices

The Don’t Be Evil button developers call the autocomplete suggestions “typeahead” results. When you use their button, the typeahead suggestion is changed. For Sandberg, her Facebook profile is shown:

The same logic I explained above is used to do this. Whatever Google lists as the first social network for a search on Sandberg’s name, that profile is used to go into the autocomplete box (though for me, it should be her CrunchBase profile there, not her Facebook profile).

I’ve covered this in my earlier articles, but I’ll stress it again. Pointing to Google+ profiles — and no other profiles — was a significant change to favor Google+ than how Google operated in the past. Consider this:

That’s where I’m starting a search for “facebook.” I’ve only entered the “face” part and, already, Google suggests that I go directly to the Facebook home page from within the search box itself.

Google started offering these types of suggestions back in 2008 and 2009. You can still see it working today, if you disable Google Instant predictions within your search settings. Typing in Amazon, or Best Buy or New York Times brings up other examples of this.

Most people have Google Instant on, since it’s the default, so they never see these type of “old” web link suggestions. Instead, they get the new Google+ profile ones. Google didn’t have to only push these, but it chose to.

Problem: Sitelinks Only For Google+

Ever done a search and noticed a bunch of listings “indented” under the main listing? Those are called sitelinks. For listings associated with Google+ accounts, they pick up special sitelinks, their last two posts on Google Plus. Here an example for TV Guide, what someone who is logged out of Google would see:

The two arrows point to the special Google+ sitelinks that are appearing. This type of thing existed before Search Plus Your World rolled out, though I can’t recall any formal announcement of it. Certainly when Direct Connect for Google+ came out, there was nothing that highlighted these sitelinks as part of that.

In fact, the addition of them feels very odd, given how just over a year ago, Google was showing sitelinks of a different sort for other social networks. For example, here’s how a search for Twitter was suggesting people our news editor Barry Schwartz should follow:

Solution: New Social Sitelinks

The Don’t Be Evil button replaces the Google+ sitelinks similarly to how other changes are done. Here’s TV Guide again:

Rather than two Google+ sitelinks being shown, only a single one appears. In this case, it’s a link to TV Guide’s Twitter profile, since that’s the first profile that the tool finds when doing a Google search for “tv guide.” A link to TV Guide’s Facebook profile follows as an alternative.

Which Way Forward?

As I’ve explained in many articles over the past two weeks, I’ve been pretty disturbed by Google’s promotion of Google+ within its search pages.

The “People & Pages” results, if they’re intended as a promotion for Google+, still come across as if they are regular search results — results that I think Google users assume come from across the web. They do not. The come only from Google+ itself.

You could argue with better labeling, that type of promotion might work better. But that still leaves questions about why Google+ profiles are being pushed within Google’s search suggestions, rather than links to other pages from across the web. The links shown there aren’t necessarily the most relevant suggestions to be shown.

Google, in particular its executive chairman Eric Schmidt, has argued that it doesn’t have all the data it needs to include other social services in the way it does for Google Plus. The failure to reach a deal with Facebook; the failure to renew a deal with Twitter, these have prevented the social signals it needs from being used, Google has said.

The Don’t Be Evil tool demonstrates this isn’t the case, that there are social signals that Google already can find which could be used, if it wants.

Of course, if Google had done so what the Don’t Be Evil tool is doing now — incorporated links to Facebook and Twitter after they’ve rejected deals — potentially Google could have been accused of scraping content from them without permission. Indeed, that was one of the things Facebook suggested last year in what was widely seen as a smear campaign.

I put that to Facebook when I talked with the company last week about the tool. I was told Facebook saw a difference between taking public signals to build a competing social service — as it felt Google was doing — and using those signals as part of a search service that points outward.

I was also assured that if Google wanted to do the types of things that the Don’t Be Evil tool is now doing, Facebook wouldn’t complain.

We’ll see. I’ll be following up with Google. In the meantime, you can read more at the Focus On The User web site, and there’s also an overview video available:

Click here to view the embedded video.

John Battelle also has a nice write-up with his perspective here, and Techmeme has a round-up of coverage from others.

Google’s Unusual Balancing Act

It can seem confusing — even hypocritical — for services like Facebook and Twitter to not release their public data more fully to anyone (including Google) without demanding deals, then complain they don’t feel represented enough in search results.

It can also seem odd that both of those companies might favor themselves in their own social networks, not include Google+, yet demand that Google somehow be “fair” to them.

The challenge Google faces, however, is that it operates both a social network and a search engine. The Google+ social network does compete with Facebook and Twitter, and anyone would expect that it should favor itself.

The Google search engine, however, was founded on the idea that it should be fair to all. That is, after all, what a good search engine should do — provide the best results, not the results best for its parent company. The continued Google+ification of Google’s search results is bringing that fairness into question.

Potentially, that’s an anti-trust issue (and Facebook, as the dominant social service, might find itself similarly under pressure to open up to competitors). But anti-trust issues aside, it’s simply a trust issue. Will users trust that Google’s results are doing the right thing by them?

I strongly encourage anyone feeling lost in all this debate and some of the tricky issues it presents to see my post from last week, FAQ: What’s The Debate About Google’s Search Plus Your World?

Finally, you can expect we’ll be taking a close look at the huge changes that Google+ is unleashing in the search marketing space at our upcoming SMX West conference in San Jose Feb. 28-March 1, in particular during these sessions:

• Getting Personal, Part 1: How Google & Bing Personalize With Social Connections
• Getting Personal, Part 2: How Google & Bing Personalize With Search History & Geography
• SEO For Google+ & Google Search
• Ask The Search Engines – Open Q&A Forum

Be sure to see the full agenda and register soon to save the most on a ticket.

Postscript: See A Proposal For Social Network Détente for some further reflections on moving ahead from today’s events.

Related Articles

• Google’s Results Get More Personal With “Search Plus Your World”
• Search Engines Should Be Like Santa From “Miracle On 34th Street”
• To Understand Google Favoritism, Think “If Google+ Were YouTube”
• Real-Life Examples Of How Google’s “Search Plus” Pushes Google+ Over Relevancy
• Pimping Google+ In Search Works: Lady Gaga Finally Joins Google+
• Larry Page Ousts Mark Zuckerberg As Most Popular On Google+
• How Google Instant’s Autocomplete Suggestions Work
• How Facebook Enables The Google Social “Scraping” It’s Upset About
• Schmidt: Google+ Not Favored, Happy To Talk Twitter & Facebook Integration
• Google Now Forcing All New Users To Create Google+ Enabled Accounts
• “Ask On Google+” Links Appearing In Google’s Search Results
• Google To Close Social Graph API, Not OpenSocial
• FAQ: What’s The Debate About Google’s Search Plus Your World?

Beyond unethical there’s a question about the legality of the behavior if proven to be true. Google says it’s investigating and sent us this statement:

These are clearly very serious allegations, and we are doing everything possible to investigate them.

(See updated statement from Google below.)

While the evidence presented by Magdalinski is very detailed and seems credible, I still have some trouble believing that Google would officially authorize a systematic campaign of fraud like this. Others have no trouble believing it and accept the allegations as true –  because many of those people believe Google has crossed over to the dark side.

One executive I spoke with the other day about something largely unrelated to Google told me he thought that Google would emerge as, “The most evil company the world has ever known.” That’s a verbatim statement. It’s pretty amazing and extreme but one hears these sorts of things now. This kind of venom used to be reserved almost exclusively for Microsoft in its antitrust heyday (e.g., “the evil empire”).

Google has in the past couple of years (and especially very recently) become a polarizing company. Among tech insiders, increasingly it seems you’re either a critic or a fan. I try to be objective in my view of Google (although I’ve been accused of being a “fanboy”). But I do tend to give the company the benefit of the doubt, unless the evidence suggests otherwise.

The Kenya controversy and the Search Plus Your World (SPYW) uproar have emerged as a sort of Rorschach test.

Google says that it intends to include much more social content into SPYW over time, including Twitter and Facebook, if those services will allow. But many people see Google simply promoting Google+ at the expense of others and tend to be extremely cynical about Google’s motives and behavior. (I find it strange that Google would so nakedly attempt to push Google+ given how directly that plays into its critics’ hands.)

The same is true this morning with the Kenya controversy. Some people simply accept that Google is guilty because they now view Google as the type of company willing to do the kinds of things claimed by Mocality. However I think there needs to be more information before anyone can reasonably conclude what happened.

The larger point here, however, is that people are increasingly inclined to leap to conclusions about Google based on their fundamental belief that the company is good or “evil.” More and more they project on to the Google ink blot whatever they want to see.

Postscript: Google apparently discovered the Mocality allegations to be accurate and issued the following statement, by Nelson Mattos, Vice-President for Product and Engineering, Europe and Emerging Markets:

“We were mortified to learn that a team of people working on a Google project improperly used Mocality’s data and misrepresented our relationship with Mocality to encourage customers to create new websites. We’ve already unreservedly apologized to Mocality. We’re still investigating exactly how this happened, and as soon as we have all the facts, we’ll be taking the appropriate action with the people involved.”

Also see our A “Mortified” Google Apologizes To Mocality For Poaching Customers, Lying About Relationship story.

Ink blot image via Shutterstock

Vertical Search Is What Search Engines Should Do

Long-time Google-critic and occasional Microsoft consultant Ben Edelman has this out today:

I’ve found more than a dozen Google services receiving favored placement in Google search results. Consider Google Blog Search, Google Book Search, Google Checkout, Google Health, Google Images, Google Maps, Google News, Google Realtime, Google Shopping, and Google Video. Some have developed into solid products with loyal users. Others are far weaker. But each enjoys a level of favored placement in Google search results that other services can only dream of.

That complaint about vertical search favoritism is the type of thing I previous said I found laughable from Google critics (though that doesn’t mean everything Edelman says is laughable, nor should be dismissed just because of his Microsoft connections).

Providing vertical search results that lead OUT of your web site is exactly what search engines should do. That’s why Bing does it, as well.

To understand more, read these past posts from me:

• The Incredible Stupidity Of Investigating Google For Acting Like A Search Engine
• The New York Times Algorithm & Why It Needs Government Regulation
• Once Again: Should Google Be Allowed To Send Itself Traffic?
• Bing’s Travel Search & Kayak Favoritism Angers No One, While Google’s Gets Headline Attention From WSJ
• Dear Congress: It’s Not OK Not To Know How Search Engines Work, Either
• Search Engines Should Be Like Santa From “Miracle On 34th Street”

Especially read the last two, if you really care about understanding this important issue.

Google+ Is Not Vertical Search

Google+ is a completely different creature. Google+ is a Google destination, a place people go not to search and exit Google but to hang around.

Yes, plenty may leave through shared links. But Google+ is not a search engine, nor was it designed to be a way to search through all the web’s socially shared content.

Google did have something like this. It was Google Realtime Search, which Google decided it had to close when it failed to renew an agreement with Twitter for however that failed to happened. See these articles for more about that:

• Google Launches Real Time Search Results
• As Deal With Twitter Expires, Google Realtime Search Goes Offline
• Google Realtime Search & The Aftermath Of The Google-Twitter Split
• Twitter Renews Deal With Bing; Google Deal Remains MIA

Google+ Is First Non-Search Engine With “Results” In Google

Since Google+ it’s not a search engine, having Google+ suggestions positioned on the Google search results pages is simply unprecedented in my time covering the company — and I’ve covered it from the start.

I cannot recall any other product where Google has done this. The search results page has always been for showing search results that come from a diverse list of sources, with the exception of maybe Google Books, where it’s hard to find similar book search engines that should be included

To really understand the big change that has happened here, how it is far different from what critics have mistakenly considered some type of unfair favoritism in the past, consider this.

Google Video Vs. YouTube

When Google launched Google Universal Search in 2008, it took great care to stress that video content blended into its results came from Google Video, not from YouTube. That’s because YouTube doesn’t have video from across the entire web. Google Video was a more inclusive service.

Now imagine if this week, Google had announced that YouTube results would replace the Google Video results displayed when you do a search. Suddenly, video content from across the web would have no visibility.

Google+ Results = YouTube, Not Google Video

That’s what Google did with the Google+ results that now get shown. It effectively launched a “who to follow” search engine, a way to show people and companies with social accounts that searchers might be interested in. But it based that solely around Google+, when it has the data to include social accounts from Twitter and Facebook, as well.

You can easily see this. Do a search that triggers the Google+ suggestions to come up, such as “music.” Click on the “see more” link at the bottom of those results. You’ll be taken into results from Google+ itself, like this.

Google+ equals YouTube, not Google Video, not a way to search beyond Google’s own hosted content.

Yes, it can seem like YouTube is favored by Google (especially more and more) in Google’s search results, but much of that is down to YouTube simply having so much content. It drowns out other things, sometimes rightfully so.

So Build A Real “Who To Follow” Search Engine

Google+ is a different issue. It’s not drowning out the others, because the others aren’t even being allowed to swim in the pool.

I’d love to see is Google retool the social suggestions that come up, so they aren’t simply Google+ “People & Pages” results injected into Google but more inclusive of other social sites, as well. Because that type of inclusive search product is what Google does well, and what we expect for the company to provide.

Related Articles

• Google’s Results Get More Personal With “Search Plus Your World”
• Twitter: Google+ Integration In Google Search Is “Bad” For Everyone
• Schmidt: Google+ Not Favored, Happy To Talk Twitter & Facebook Integration
• Twitter Cries Foul Over Google “@WWE” Search, But Google Still Beats Bing
• Search Engines Should Be Like Santa From “Miracle On 34th Street”
• Real-Life Examples Of How Google’s “Search Plus” Pushes Google+ Over Relevancy

Imagine, Sending People To Other Stores!

One of my favorite movies is “Miracle On 34th Street.” The original, not the remake, by the way. For those not familiar, Kris Kringle — the real Santa Claus — gets hired to play the Macy’s Santa Claus.

He nearly gets fired when it’s discovered that he’s sending people away to other stores rather than Macy’s, if other stores have the right gift for a child or a better price. But then Macy’s realizes the shoppers love its store even more for the “send away” policy.

One woman in particular says:

Listen. I want to congratulate you and Macy’s on this wonderful new stunt you’re pulling. Imagine, sending people to other stores. I don’t get it… Imagine a big outfit like Macy’s putting the spirit of Christmas ahead of the commercial. It’s wonderful.

Search Engine Santas

That’s how search engines are supposed to be. They’re supposed to send you away to the best information, even if they don’t have their own in stock.

In fact, at first, search engines didn’t even have anything “in stock” of their own to send people to. When some search engines like Yahoo or Lycos became portals in the late 1990s, we started seeing a “Search Santa” that wasn’t so generous, that didn’t want the shoppers to leave.

Google’s Has Been A Stellar Santa

Google has largely held up to the Miracle On 34th Street model. The charges that Google favors itself have come from an incredibly tiny number of competitors, some of whom complain they’re not being favored enough when they get three-quarters of ALL their traffic from Google.

Hey, that’s not Google competing. That’s Google ushering huge numbers of its own potential “shoppers” out the door to Gimbel’s.

Most of these charges I find laughable. Seriously laughable. Quite frankly, if Google never existed, and the dominant search player that had grown up was Yahoo or Microsoft, there’s an excellent chance that people would be paying through the nose just for the “right” to be included for a chance to rank well.

People who don’t understand this simply have no idea of history in the search space, no memory of the dark days of paid inclusion. Google has been an exceptionally good actor given the dominance it has in the search space.

If you really don’t believe it, go check the traffic you receive from search engines. Is 65% of it coming from Google versus other search engines? Chances are, you’re getting more than that, despite that being Google’s share of the search market. For that to happen, it means the other search engines are recycling more “shoppers” back to their own stores than Google does

If you really care about understanding these facts. If you really want to have an open mind, to form an opinion out of evidence and rational thought rather than knee-jerk anti-Google hatred, check out the articles I’ve listed at the end in the anti-trust section for some really detailed send-ups of anti-trust claims.

But Google’s Showing A Different Side, Today

Today is different. Today’s change is one of the few times where I’m thinking “What the hell are you doing, Google?” Anyone familiar with my writings, I’d like to believe, knows that when I ring an alarm bell, there’s a really good reason to start heading toward the fire exits.

I’m not alone. Twitter’s already sent out a statement over the move:

For years, people have relied on Google to deliver the most relevant results anytime they wanted to find something on the Internet.

Often, they want to know more about world events and breaking news. Twitter has emerged as a vital source of this real-time information, with more than 100 million users sending 250 million Tweets every day on virtually every topic. As we’ve seen time and time again, news breaks first on Twitter; as a result, Twitter accounts and Tweets are often the most relevant results.

We’re concerned that as a result of Google’s changes, finding this information will be much harder for everyone. We think that’s bad for people, publishers, news organizations and Twitter users.

MG Siegler also gives Google a hard rap on the head, as well, in his post. There will be more of this to come.

On The Matter Of Anti-Trust, The Court Will Hear Evidence….

There are two areas of possible dispute:

1. Google is only listing information from its Google+ posts
2. Google is offering suggested users to follow from its Google+ network

The first is debatable. Google is prevented for various reasons from accessing privately or less-than-publicly (let’s call this “limited”) information shared at places like Facebook, Twitter, Flickr and others.

Some of these reasons aren’t in Google’s control. Does anyone really believe that Facebook decided to let Bing have its data but not Google because Mark Zuckerberg thought an “underdog” would do better with it, as he said at the time?

In Twitter’s case, it generates so much data that it simply cannot be gathered up by Google in normal ways. If Twitter wants to be fully indexed by Google in the way its statement claims, it need to provide a firehose of data to Google.

Google would gladly take it, I’m sure. But my guess is that Google no longer wants to pay for it, much less carry Twitter’s ads on its own site, both things that were part of its previous deal with Twitter and virtually unprecedented for Google to do for anyone.

We don’t know all the details, who’s to blame for what and in what proportions for all of this. We do know that both Facebook and Twitter currently get plenty of traffic from Google even without formal cooperation.

This Isn’t Santa-Like

The second is cause for serious concern. There’s no reason that a search for something like “music” should give special positioning to suggest people follow social media accounts from Britney Spears, Alicia Keys or Snoop Dogg, just because those accounts are on Google Plus:

If these artists are on other social networks (and I’m sure they all are), those should be suggested as well. If Google’s in the business of now recommending people to follow socially, then as a Search Santa, it should be sending people to wherever the best places are. Even if they’re not at Google.

That, by the way, is equally true for Bing. Ironically, it kicked out a bunch of merchants on what seems pretty flimsy grounds just before Black Friday and Cyber Monday. That felt very much like it wasn’t playing the Search Santa role it’s supposed to, as well.

NOTE: Miracle On 34th Street imagine blatantly taken from IMDb. I hope they’ll forgive me. Read about the movie there.

Postscript: See:

• Schmidt: Google+ Not Favored, Happy To Talk Twitter & Facebook Integration
• Real-Life Examples Of How Google’s “Search Plus” Pushes Google+ Over Relevancy
• To Understand Google Favoritism, Think “If Google+ Were YouTube”

For further follow-ups on the Twitter-Google dispute.

Related: Facebook, Twitter & Google Data War

• Facebook On Social Search: ‘We Want To Work With Everybody’, Oct. 2010
• Facebook: You’ve No Right To Export Email Addresses (Unless It’s To Yahoo & Microsoft), Nov. 2010
• Google & Facebook: If You’re So Smart, Work It Out!, Nov. 2010
• Examining Facebook’s “Smear Campaign” Concerns About Google Social Circle, May 2011
• How Facebook Enables The Google Social “Scraping” It’s Upset About, May 2011
• Google Realtime Search & The Aftermath Of The Google-Twitter Split, July 2011

Related: Facebook & Bing

• Bing, Now With Extra Facebook: See What Your Friends Like & People Search Results, Oct. 2010
• Bing’s Facebook Fans Do 68% More Searches Than Average Bing Users, Aug. 2011
• Bing Integrates Facebook Likes Further Into Its Search Results, Feb. 2011
• Bing Ups Ante In Social Search, Adds More Facebook “Likes” To Search Results, May 2011
• Has Facebook Become The Master Key To Unlocking The Web?, May 2011
• Facebook’s Zuckerberg To Charlie Rose: “We Just Do One Thing”, Nov. 2011
• Facebook Timeline Officially Released To The Public, Dec. 2011

Related: Google Anti-Trust

• Googleopoly: The Definitive Guide To Antitrust Investigations Against Google, June 2011
• Google Senate Hearings: The Post-Game Show, Sept, 2011
• Dear Congress: It’s Not OK Not To Know How Search Engines Work, Either, Dec. 2011
• Bing’s Travel Search & Kayak Favoritism Angers No One, While Google’s Gets Headline Attention From WSJ, Dec. 2011

Where We Came From

Some might believe that search engines hate SEOs, hate publishers and have done little over the years to help them. They are mistaken, either choosing to deliberately ignore the gains or, more likely, are simply unaware of how far things have come.

When I first started writing about SEO issues nearly 16 years ago, in 1996, we had little publisher support beyond add URL forms. Today, we have entire toolsets like Google Webmaster Central and Bing Webmaster Tools, along with standalone features and options, which allow and provide:

• Ability to submit & validate XML sitemaps
• Ability to view crawling & indexing errors
• Ability to create “rich” listings & manage sitelinks
• Ability to migrate a domain
• Ability to indicate a canonical URL or preferred domain
• Ability to set crawl rates
• Ability to manage URL parameters
• Ability to view detailed linkage information to your site
• Ability to view keywords used to reach your site
• Notifications of malware or spam issues with your site

There’s even more beyond what I’ve listed above. The support publishers enjoy today was simply unimaginable to many veteran SEOs who were working in the space a decade ago.

The advancement has been welcomed. It has helped publishers better manage their placement in those important venues of the web, the search engines. It has helped search engines with errors and problems that would hurt their usability and relevancy.

That’s why 2011 was so alarming to me. After years of moving forward, the search engines took a big step back.

The Loss Of Link Data

One of the most important ways that search engines determine the relevancy of a web page is through link analysis. This means examining who links to a page and what the text of the link — the anchor text — says about the page.

However, for years Google has deliberately suppressed the ability for outsiders to see what links tell it about any particular page. Want to know why THAT result shows up for Santorum? Why Google was returning THAT result for “define English person” searches? Sorry.

Google won’t help you understand how links have caused these things. It refuses to show all the links to a particular page, or the words used within those links to describe a page, unless you are the page’s owner.

Why? Google’s rationale has been that providing this information would make it harder for it to fight spam. Potentially, bad actors might figure out some killer linking strategy by using Google’s own link reporting against it.

It’s a poor argument. Despite withholding link data, it’s painfully easy to demonstrate how sites can gain good rankings in Google for competitive terms such as “SEO” itself by simply dropping links into forums, onto client pages or into blog templates.

Given this, it’s hard to understand what Google thinks it’s really protecting by concealing the data. But until 2011, there was an easy alternative. Publishers and others could turn to Google-rival Yahoo to discover how people might be linking to a page.

Goodbye Yahoo Site Explorer

Yahoo launched its “Yahoo Site Explorer” back in September 2005, both as part as a publicity push to win people away from Google and to provide information to publishers. The tool allowed anyone to see what link data Yahoo had about any page in its listings.

Today, Yahoo still supposedly wants to win people away from Google. But because Yahoo’s web search results are now powered by Bing, Yahoo has little reason to provide tools to support publishers. That’s effectively Bing’s problem now.

Yahoo closed Yahoo Site Explorer at the end of last November, saying as it still does on the site now:

Yahoo! Search has merged Site Explorer into Bing Webmaster Tools. Webmasters should now be using the Bing Webmaster Tools to ensure that their websites continue to get high quality organic search traffic from Bing and Yahoo!.

That’s not true. Yahoo Site Explorer was not merged into Bing Webmaster Tools. It was simply closed. Bing Webmaster Tools doesn’t provide the ability to check on the backlinks to any page in the way that Yahoo Site Explorer allowed.

The closure supposedly came after Yahoo “listened to your feedback” about what publishers wanted, as it posted earlier this year. I don’t know what feedback Yahoo was hearing, but what I’ve heard has been people desperately pleading with Yahoo or Bing to maintain the same exact features that Yahoo Site Explorer provided — and pleading for well over a year.

Yahoo-Bing Deal Has Reduced Competition & Features

When the US Department Of Justice granted its approval for Yahoo to partner with Microsoft, that was supposed to ensure that the search space stayed competitive. From what the Department Of Justice said in 2010:

After a thorough review of the evidence, the division has determined that the proposed transaction is not likely to substantially lessen competition in the United States, and therefore is not likely to harm the users of Internet search, paid search advertisers, Internet publishers, or distributors of search and paid search advertising technology.

I’d say dropping Yahoo Site Explorer did harm to both users of internet search and internet publishers. Yahoo Site Explorer was a distinctive tool that only Yahoo offered, allowing both parties named by the DOJ to better understand the inner workings of the search engines they depend on. It also reduced competitive pressure for Google to offer its own tool.

Indeed, things have gotten worse since Yahoo Site Explorer closed. At the end of last December, Bing officially confirmed that it no longer supports the link command in its help forum.

Next To Go, The Link Command?

The link command allows you to enter any page’s web address prefaced by “link:” in order to find links that point at that page. It’s a long-standing command that has worked for many major search engines as far back to late 1995, when AltaVista launched.

Google still supports this command to show some (but not all) of the links it knows about that point at pages. I’d link to Google’s documentation of this, but the company quietly dropped that some time around May 2008. Here’s what it used to say:

Here’s how the command still works at Google. Below, I used it to see what links Google says point to the home page of the official Rick Santorum campaign web site:

The first arrow shows you how the command is being used. The second arrow shows you how Google is reporting there are 111 links pointing to the page. Imagine that. Rick Santorum, currently a major Republican candidate for US president, and Google says only 111 other pages link to his web site’s home page.

The reality is that many more pages probably link over. Google’s counting them but not showing the total number to people who care about what exactly is being considered. I’ll demonstrate this more in a moment, but look at the worse situation on Bing:

One link. That’s all Bing reports that it knows about to those in the general public who may care to discover how many links are pointing to the Rick Santorum web site.

It’s Not Just An SEO Thing

People do care, believe me. I actually started writing this article last Monday and got interrupted when I had to cover how Google might have been involved with a link buying scheme to help its Chrome browser rank better in Google’s own search results.

I doubted that was really the main intent of the marketing campaign that Google authorized (Google did err on the side of caution and punished itself), but the lack of decent link reporting tools from Google itself left me unable to fully assess this as an independent third-party.

As soon as that story was over, renewed attention was focused on why Rick Santorum’s campaign web site wasn’t outranking a long-standing anti-Santorum web site that defines “santorum” as a by-product of anal sex.

Major media outlets were all over that story. My analysis was cited by The Economist, CNN, The Telegraph, The New York Times, MSNBC and Marketplace, to name only some.

But again, I — or anyone who really cared — was unable to see the full links that Google knew about pointing at both sites, much less the crucial anchor text that people were using to describe those sites. Only Google really knew what Google knew.

Third Party Options Good But Not The Solution

If you haven’t heard more complaints over the closure of Yahoo Site Explorer, and the pullback on link data in general, that’s because there are third-party alternatives such as Majestic Site Explorer or the tool I often use, SEOmoz’s Open Site Explorer.

These tools highlight just how little the search engines themselves show you. Consider this backlink report from Open Site Explorer for the Rick Santorum campaign’s home page:

The first arrow shows how 3,581 links are seen pointing at the page. Remember Google, reporting only 111? Or Bing, reporting only 1?

The next two arrows show the “external” links pointing at both the Santorum home page and the anti-Santorum home page. These are links from outsiders, pointing at each page. You can see that the anti-Santorum page has four times as many links pointing at it than the Santorum campaign page, a clue as to why it does so much better for a search on “santorum.”

But it’s not just number of links. Using other reports, I can see that thousands of links leading to both sites have the text “santorum” in the links themselves, which is why they both are in the top results for that word.

Because the anti-site has so many more links that say “santorum” and “spreading santorum,” that probably helps it outrank the campaign site on the single word. But because the official site has a healthy number from sources including places like the BBC saying “rick santorum” in the links, that — along with its domain name of ricksantorum.com — might help it rank better for “rick santorum.”

It’s nice that I can use a third party tool to perform this type of analysis, but I shouldn’t have to. It’s simply crazy — and wrong — that both Google and Bing send searchers and publishers away from their own search engines to understand this.

For one, the third party tools don’t actually know exactly what the search engines themselves are counting as links. They’re making their own estimates based on their own crawls of the web, but that doesn’t exactly match what Google and Bing  know (though it can be pretty good).

Not Listing Links Is Like Not Listing Ingredients

For another, the search engines should simply be telling people directly what they count. Links are a core part of the “ingredients” used to create the search engine’s results. If someone wants to know if those search results are healthy eating, then the ingredients should be shared.

Yes, Google and Bing will both report link data about a publisher’s own registered site. But it’s time for both of them to let anyone look up link data about any site.

The Blekko search engine does this, allowing anyone logged in to see the backlinks to a listed page. Heck, Blekko will even give you a badge you can place on your page to show off your links, just as Yahoo used to. But for Google, it’s “normal” for its link command to not show all the links to a page. Seriously, that’s what Google’s help page says.

Google, in particular, has made much of wanting people to report spam found in its search results. If it really wants that type of help, then it needs to ensure SEOs have better tools to diagnose the spam. That means providing link data for any URL, along with anchor text reporting.

Google has also made much about the need for companies to be open, in particular pushing for the idea that social connection should be visible. Google has wanted that, because until Google+ was launched, Google had a tough time seeing the type of social connections that Facebook knew about.

Links are effectively the social connections that Google measures between pages. If social connections should be shared with the world, then Google should be sharing link connections too, rather than coming off as hypocritical.

Finally, it doesn’t matter if only a tiny number of Google or Bing users want to do this type of link analysis. That’s often the pushback when this issue comes up, that so few do these type of requests.

Relatively few people might read the ingredients labels on the food they eat. But for the few that do, or for anyone who suddenly decides they want to know more, that label should be provided. So, too, should Google and Bing provide link data about any site.

Goodbye Keyword Referrer Data

While I’m concerned about the pullback on link data, I’m more concerned about how last October, Google stopped reporting to publishers the keywords people used to find their web sites, for times when those people were logged into Google.

Link data has long been suppressed by Google. Holding back on keyword data is a new encroachment.

Google has said this was done to protect user privacy. I have no doubt many in the company honestly believe this. But it if was really meant to protect privacy, then Google shouldn’t have deliberately left open a giant hole that continues to provide this data to its paid advertisers.

Worse, if Google were really serious about protecting the privacy of search terms, then it would disable the passing of referrers in its Chrome browser. That hasn’t happened.

Unlike the long examination of link data above, I’ll be far more brief about the situation with Google withholding link data. That’s because I’ve already written over 3,000 words looking at the situation in depth last October, and that still holds up. So please see my previous article, Google Puts A Price On Privacy, to understand more.

Google’s Weak Defense

Since my October story, the best defense that Google’s been able to concoct for withholding keyword data from non-advertisers is a convoluted, far-fetched argument that makes its case worse, not better.

Google says that potentially, advertisers might buy ads for so many different keywords that even if referrer data was also blocked for them, the advertisers could still learn what terms were searched for by looking through their AdWords campaign records.

For example, let’s say someone did a search on Google for “Travenor Johannisoon income tax evasion settlement.” I’ve made this up. As I write this, there are no web pages matching a Google search for “Travenor Johannisoon” at all. But…

• If this were a real person, and
• someone did that search, and
• if a page appeared in Google’s results, and
• someone clicked on that page…

then the search terms would be passed along to the web site hosting the page.

Potentially, this could reveal to a publisher looking at their web analytics that there might be a settlement for income tax evasion for involving a “Travenor Johannisoon.” If the publisher starting poking around, perhaps they might uncover this type of information.

Of course, it could be that there is no such settlement at all. Maybe it’s just a rumor. Anyone can search for anything which doesn’t make it into a fact.

More likely, the search terms are so buried in all the web analytics data that the site normally receives that this particular search isn’t noticed at all, much less investigated.

Extra Safe Isn’t Extra Safe

Still, to be extra safe, Google has stopped passing along keyword data when people are signed-in. Stopped, except to its advertisers. Like I said, Google argues that potentially advertisers might still see this information even if they were also blocked.

For instance, say someone runs an ad matching any searches with “income tax evasion” in them. If someone clicked on the ad after doing a search for “Travenor Johannisoon income tax evasion settlement,” those terms would be passed along though the AdWords system to the advertiser, even though the referrer might pass nothing to the advertiser’s web analytics system.

So, why bother blocking?

Yes, this could happen. But if the point is to make things more private, then blocking referrers for both advertisers and non-advertisers would still make things harder. Indeed, Google still has other “holes” where “Travenor Johannisoon” might find his privacy exposed just as happens potentially with AdWords.

For example, if someone did enough searches on the topic of Travenor and tax evasion, that might cause it to appear one of Google Instant’s suggested searches.

So why bother blocking?

Also, while Google blocks search terms from logged-in users in referrer data, those same searches are not blocked from the keyword data it reports to publishers through Google Webmaster Central. That means the Travenor search terms could show up there.

So why bother blocking?

Nothing has changed my view that, despite Google’s good intentions, its policy of blocking referrers only for non-advertisers is incredibly hypocritical. Google purports this is done to protect privacy, but it puts its own needs and advertisers desires above privacy.

Blocking referrers is a completely separate issue from encrypting the search results themselves. That’s good and should be continued. But Google is deliberately breaking how such encryption works to pass along referrer data to its advertisers. Instead, Google should block them for everyone or block them for no one. Don’t play favorites with your advertisers.

What Google & Bing Should Do

Made it this far? Then here’s the recap and action items for moving forward.

Bing should restore its link command, if not create a new Bing Site Explorer. Google should make sure that its link command reports links fully and consider its own version of a Google Site Explorer. With both, the ability for anchor text reports about any site is a must.

If there are concerns about scraping or server load, make these tools you can only use when logged in. But Yahoo managed to provide such a tool. Blekko is providing such statistics. Tiny third-party companies are doing it. The major search engines can handle it.

As for the referrer data, Google needs to immediately expand the amount of data that Google Webmaster Central reports. Currently, up to 10,000 terms (Google says up to 1,000, but we believe that’s wrong) for the past 30 days are shown.

In November, the head of Google’s spam team Matt Cutts — who’s also been involved with the encryption process — said at the Pubcon conference that Google is considering expanding the time period to 60 days or the queries to 2,000 (as said, we think — heck, we can see, they already provide more than this). Slightly more people wanted more time than more keywords shown.

I think Google should do more than 60 days. I think it should be providing continuous reporting and holding that data historically on behalf of sites, if it’s going to block referrers. Google is already destroying historical benchmarks that publishers have maintained. Google’s already allowed data to be lost for those publishers, because they didn’t begin to go in each day and download the latest information.

So far, all Google’s done is provide an Python script to make downloading easier. That’s not enough. Google should provide historical data, covering a big chunk of the terms that a site receives. It’s the right thing to do, and it should have been done already.

What Publishers Can Do

An anti-SOPA-like effort as targeted GoDaddy isn’t going to work with the search engines. That’s because the two biggest things that publishers could “transfer” out of Google and Bing are their ads and their web sites. But there’s no place to transfer these to that wouldn’t hurt the publishers with incredible amounts of lost traffic.

This doesn’t mean that publishers are powerless, however.

Bing is desperate to be seen as the “good” search engine against “evil” Google. Publishers should, whenever relevant, remind Bing that it’s pretty evil not to have maintained its own version of Yahoo Site Explorer much less to have closed the link command.

Mention it in blog posts. Mention it in tweets. Bring it up at conferences. Don’t let it die. Ask Bing why it can’t do what little Blekko can.

As for Google, pressure over link data is probably best expressed in terms of relevancy. Why is Google deliberately preventing this type of information from being studied? Is it more afraid that doing so will reveal weaknesses in its relevancy, rather than potential spam issues? Change the debate to relevancy, and that gets Google’s attention — plus the attention of non-publishers.

There’s also the issue of openness. Google shouldn’t be allowed to preach being “open” selectively, staying closed when it suits Google, without some really good arguments for remaining closed. On withholding link data, those “closed” arguments no longer stand up.

As for the referrer data, Google should be challenged in three ways.

First, the FTC will be talking to publishers as part of its anti-trust investgation into Google’s business practices. Publishers, if asked, should note that by withholding referrer data except for Google’s advertisers, it’s potentially harming competing retargeting services that publishers might prefer to use. Anti-trust allegations seem to really get Google’s attention, so make that wheel squeak.

Second, question why Google is deliberately leaving a privacy hole open for the searchers it’s supposedly trying to protect. If Google’s really worried about what search terms reveal, the company needs a systematic way to scrub potentially revealing queries from everything: suggested searches, reporting in Google Webmaster Central, AdWords reporting as well as referrer data.

Finally, withhold your own data. Are you opted-in to the data sharing on Google Analytics that launched back in 2008? Consider opting-out, if so:

To opt-out, when you log in, select an account, then select “Edit Analytics Account” next to the name of the account in the Overview window, then you’ll see options as shown above and as explained on this help page.

Opting out means you can’t use the benchmarking feature (fair enough, and no loss if you don’t use it) and Conversion Optimizer. If you still want Conversion Optimizer, don’t opt-out or alternatively, tell Google that you should have a choice to share data solely for use with that product but not other Google products.

There might be other drawbacks to not sharing that I’m missing. But we haven’t been sharing here at Search Engine Land since the beginning of the year. So far, we’re not having any problems.

Google loves data. Withholding your own is another way for publishers to register their displeasure about having data withheld from them. And it’s the type of thing that Google just might notice.

Related Articles

• What Is SEO / Search Engine Optimization?
• Search Engine Land’s Guide To SEO
• The Periodic Table Of SEO Ranking Factors
• What Is Google PageRank? A Guide For Searchers & Webmasters
• How The “Focus On First” Helps Hide Google’s Relevancy Problems
• Yahoo Completes Global Organic Transition To Bing
• Yahoo Site Explorer Closing Down Monday, November 21st
• The Microsoft-Yahoo Search Deal, In Simple Terms
• Google Releases New Link Reporting Tools
• Google Now Reporting Anchor Text Phrases
• Yahoo Adds Link Badge In Site Explorer
• Link Building Tool Review: Link Research Tool Set By CEMPER
• Link Building Tool Review: SEOmoz PRO
• Link Building Tool Review: SEO Book
• Link Building Tool Review: Raven Tools
• Link Building Tool Review: Majestic SEO
• Bing Webmaster Tools Launches New Link Reports; Google Webmaster Tools Changes Theirs
• Blekko’s SEO Tools: What Information Do They Provide?
• Blekko Offers New Linkroll Widget & More Publisher Tools
• Google’s Chrome Page No Longer Ranks For “Browser” After Sponsored Post Penalty
• Should Rick Santorum’s “Google Problem” Be Fixed?
• How Rick Santorum Is Making His “Google Problem” Worse
• For “Define An English Person,” Google Suggests The C-Word
• Google’s Spam Report Page Gets “Biggest Refresh” In Years
• Google: As Open As It Wants To Be
• Google’s Facebook Competitor, The Google+ Social Network, Finally Arrives
• How Google Instant’s Autocomplete Suggestions Work
• Google Webmaster Tools Adds Page-Level Query Data
• Google Webmaster Tools Search Queries Report Now Less Accurate
• Google Analytics Benchmarking Feature, Data Sharing & Audio Ad Charting
• Guest Opinion: Is Google’s Privacy Move Really An Anti-Competitive Practice?
• Dear Congress: It’s Not OK Not To Know How Search Engines Work, Either
• Googleopoly: The Definitive Guide To Antitrust Investigations Against Google
• Keyword “Not Provided” By Google Spikes, Now 7-14% In Cases
• Google Puts A Price On Privacy