A recently enacted law seeks to remove all such content from the internet in India. Facebook, Google, Yahoo and Microsoft were sued under the law and had been arguing in Indian court that it was all but impossible for them to comply as a practical matter. The law makes online publishers potentially liable for the acts of individual users and third parties (think “offensive” blog hosted on Blogger or “objectionable” video uploaded to YouTube).

According to a BBC report Indian Communications Minister Kapil Sibal said the following in December about the desired impact of the new law:

My aim is that insulting material never gets uploaded. We will evolve guidelines and mechanisms to deal with the issue. [The companies] will have to give us the data, where these images are being uploaded and who is doing it.

Google, Facebook and others had argued that they had no control over individuals and should not be held liable for their conduct accordingly. They said they cannot “pre-filter” material generated by millions of users. This is what would be called a “prior restraint” against free speech in the US.

Indian courts have been unsympathetic and demanded that the companies comply or be blocked entirely “like in China.” The BBC says that the companies have now complied and removed offending material at issue in a particular civil lawsuit.

However the overly vague nature of the statute on which these claims are based almost guarantees that Google, Yahoo, Facebook, Microsoft and other online publishers in India will face a steady stream of litigation from individuals or groups “offended” by this or that image, article or video.

Stock image used under license from Shutterstock

Related Entries

• Free Speech Battle In India: Google, Facebook Summoned By Court Over “Inflammatory Images”
• India Set To Bring Heavy Hand Of Censorship Down On Facebook, Google
• EU Ready To Impose Tough Privacy Rules On Google, Facebook
• US Government Makes Most Content Removal Requests To Google So Far in 2011

While some sites like Wikipedia are going black for the day on Wednesday, Google has instead chosen to cover its logo in black and add a short message on its home page: Tell Congress: Please don’t censor the web!

The message links to a new End Piracy, Not Liberty page where Google is encouraging visitors to sign a petition against the legislation and sharing this message:

Two bills before Congress, known as the Protect IP Act (PIPA) in the Senate and the Stop Online Piracy Act (SOPA) in the House, would censor the Web and impose harmful regulations on American business. Millions of Internet users and entrepreneurs already oppose SOPA and PIPA.

The Senate will begin voting on January 24th. Please let them know how you feel. Sign this petition urging Congress to vote NO on PIPA and SOPA before it is too late.

The page also includes a PDF/infographic detailing the groundswell of opposition to SOPA and PIPA.

If you’re not familiar with SOPA (and its Senate counterpart, PIPA), see Chris Sherman’s article on Marketing Land, What All Marketers Need To Know About SOPA – The Stop Online Piracy Act.

What About Bing, Yahoo & Other Search Engines

Meanwhile, as of this moment, both Bing and Yahoo are carrying on in a business-as-usual mode.

Microsoft issued a statement today saying that it does oppose “the SOPA bill as currently drafted,” but neither Microsoft.com nor Bing.com make any mention right now of that opposition. Here’s the current Bing home page, which has a scenic photo of Norway.

Likewise, neither Yahoo.com nor its search-only home page at search.yahoo.com are showing anything related to the legislation or protests, unless you count the appearance of “PIPA” as No. 9 on Yahoo’s “Trending Now” topics.

There’s also nothing currently showing on Ask.com’s home page, but a PR rep for IAC notified us earlier tonight that Ask will “donate” ad space on its home page to voice its opposition to the legislation. The ad space, we’re told, will link to this OpenCongress.org page with details about SOPA and tools for people to contact their representatives.

Postscript: Although Yahoo.com isn’t showing anything related to SOPA/PIPA, Yahoo’s Flickr photo sharing site has gotten involved by letting users darken any photo on the site — their own or anyone else’s. (Photo owners can opt out of having their photos darkened.) Here’s one of my recent photos that I’ve darkened:

Postscript #2: Duck Duck Go has joined the ranks of search engines that are protesting SOPA/PIPA. Here’s what its homepage looks like today.

Related Stories

• Big List Of SOPA Links
• What All Marketers Need To Know About SOPA – The Stop Online Piracy Act
• The Stop Online Piracy Act (SOPA) Stalls In Congress
• #BlackoutSOPA: A Look At The Social Media Movement That Helped Stall The SOPA Legislation
• How To Blackout Your Site (For SOPA/PIPA) Without Hurting SEO
• Google Blackens Its Logo To Protest SOPA/PIPA, While Bing & Yahoo Carry On As Usual
• Why The Web Is Going Dark Over SOPA & PIPA

SOPA and PIPA are supported by a range of corporate entities and media companies that elicited these draconian bills from Congress to address global IP piracy, copyright violations and the “rogue” sites allegedly responsible. (See Chris Sherman’s comprehensive run-down of SOPA.)

Somewhat ironically, CNET parent CBS Corp. is among the companies supporting SOPA. Indeed, many of the news outlets reporting on SOPA have corporate owners that support the measure. The full list is a veritable “who’s who” of American corporations.

Google has been a vocal opponent of SOPA/PIPA since the beginning. As the tide has turned against SOPA/PIPA  in recent weeks, support for the bills in Congress has waned. In addition, the White House just came out against SOPA and PIPA in their current form.

Google’s homepage link will be another nail in the coffin of these dreadful bills. But while they may now seem dead, rest assured that like a zombie this legislation will probably rise from the grave again in some modified form. There are too many powerful entities that want to see this legislation pass.

A couple of years ago, Piper Jaffray analyst Gene Munster calculated the value of a hypothetical ad placement on Google’s homepage to be worth between $4 and $5 million (at a $15 or $20 CPM) if it actually had to be bought.

Postscript: See our follow-up stories:

• Why The Web Is Going Dark Over SOPA & PIPA
• Google Blackens Its Logo To Protest SOPA/PIPA, While Bing & Yahoo Carry On As Usual

(Photo of U.S. Capitol licensed under Creative Commons from Flickr user natalie419)

In case the problem with implementation of a law against “objectionable” content isn’t self-evident its provisions are vague and scope extremely broad. The law includes a laundry list of concepts and content restrictions susceptible to highly subjective interpretation:

[The law implicates] anything that is “harmful, threatening, abusive, harassing, blasphemous, defamatory, pornographic, libellous, invasive of another’s privacy, hateful, disparaging, racially, ethnically or otherwise objectionable, relating to money laundering or gambling.” 

The law, which Google sought to modify before its enactment, is part of an effort to regulate morality and content on the internet in India, a country with an otherwise healthy free-speech tradition. But this law gives enormous power to both the government and to individuals to block or remove content that is subjectively disfavored for one reason or another.

Accordingly, the ability to criticize public figures and to engage in critical public discussion or debate over history and political or social issues is directly threatened by the law. Google and other internet companies originally opposed the law for logistical and liability reasons.

With more than 1 billion people, India is one of the largest potential markets in the world for Google, Facebook and other technology companies. But together with China (the largest market), US and European companies face challenging government restrictions and content censorship.

The numbers come from Google’s latest semi-annual update to the Government Requests section of its Transparency Report. The Government Requests data represents government inquiries to Google for information about users and/or content removal requests.

Newly added in this latest release is data about the number of Google users or accounts that are specified in the government requests. For example, those 5,950 U.S. government requests specified a little more than 11,000 U.S. Google users/accounts — about 5x more than the numbers of accounts specified by India’s requests.

The Government Requests page also has some specifics about the various reasons that worldwide governments have asked Google to remove content over the past six months.

(Stock image of US flag via Shutterstock.com, used with permission.)

The company just released data that covers the period from July to December 2010. It first launched the report in 2010, covering data from 2009.

The new report format makes it easier to look at data on a country-by-country basis. It also provides more detailed information on government requests for items to be taken out of the index, describing the various circumstances that resulted in the requests. For example, in the U.S., 34 court orders resulted in the removal of 1,375 items for defamation. In response to requests from government officials, such as local police or the FBI, three items were removed for reasons of privacy and security; one item was taken down from YouTube for national security reasons; and one item was removed form YouTube because of violence.

Another new piece of information reveals the percentage of government requests for Google user data that are complied with. This varies from 94% in the United States (of 4,601 requests) to 0% in Hungary (of 68 requests).

Google also released new information on places where traffic to its servers appear to be blocked, and for how long. The report shows, for example, that traffic to Google from Lybia dropped precipitously on May 3 and has never recovered.

A popular Irish hotel has sued Google for defamation because Google’s autocomplete feature suggests to searchers that the hotel is in receivership.

Searchers looking for the Ballymascanlon Hotel — a four-star property that’s reportedly one of the most popular wedding venues in northeast Ireland and is not in financial trouble — see “ballymascanlon hotel receivership” as an autocomplete suggestion as soon as they’ve typed only eight letters of the hotel name. According to a recent Sunday Times article (quoted here by TJ McIntyre), some brides have contacted the hotel “in tears” after seeing the autocomplete suggestion, no doubt fearing that their wedding plans would have to be scrapped.

As Mark Collier writes, the hotel isn’t seeking punitive damages from Google; the suit only asks for an injunction to stop Google from showing the autocomplete suggestion about receivership, and for Google to pay the hotel’s legal fees.

Collier also details how the hotel made multiple attempts to contact Google about the issue and resolve it away from court – beginning with online channels and eventually escalating to attorney’s letters and even including the autocomplete problem in a DMCA complaint filed in March.

Previous Autocomplete Cases

Google has already faced similar complaints in other countries, and hasn’t fared well in the courts. The company lost two cases last year in France; see our articles Google Loses French Lawsuit Over Google Suggest and Google Convicted Again In France Over Google Suggest.

Earlier this year, Google also lost cases in Italy and Argentina.

How Autocomplete Works

Google has explained many time that autocomplete suggestions come from actual search activity. In Danny Sullivan’s article, How Google Instant’s Autocomplete Suggestions Work, the company commented on the Italian case I mentioned above:

We believe that Google should not be held liable for terms that appear in Autocomplete as these are predicted by computer algorithms based on searches from previous users, not by Google itself.

But Google’s argument that autocomplete suggestions are algorithmic doesn’t seem to stand up to legal scrutiny, perhaps because the company has manually removed piracy-related terms in the past, and its help pages list other cases — pornography, violence, hate speech, etc. — where suggestions will be removed.

I’m certainly not a lawyer, nor do I play one on Search Engine Land. So, whether that happens again in Ireland is anyone’s guess at this point.

(Thanks to Mark Collier for tipping us to this story. If you have news tips to share, please contact us.)

Google & Search Suggestions

Google was not the first search engine to offer search suggestions, nor it is the only one. But being the most popular search engine has caused many to look at Google’s suggestions more closely.

Google has been offering “Google Suggest” or “Autocomplete” on the Google web site since 2008 (and as an experimental feature back since 2004). So suggestions — or “predictions” as Google calls them — aren’t new.

What Google suggests for searches gained new attention after Google Instant Search was launched last year. Google Instant is a feature that automatically loads results and changes those results. That interactivity caused many to take a second look at suggestions, including an attempt to list all blocked suggestions.

Suggestions Based On Real Searches

The suggestions that Google offers all come from how people actually search. For example, type in the word “coupons,” and Google suggests:

• coupons for walmart
• coupons online
• coupons for target
• coupons for knotts scary farm

These are all real searches that have been done by other people. Popularity is a factor in what Google shows. If lots of people who start typing in “coupons” then go on to type “coupons for walmart,” that can help make “coupons for walmart” appear as a suggestion.

Google says other factors are also used to determine what to show beyond popularity. However, anything that’s suggested comes from real search activity by Google users, the company says.

Suggestions Can Vary By Region & Language

Not everyone sees the same suggestions. For example, above in the list is “coupons for knotts scary farm.” I see that, because I live near the Knott’s Berry Farm amusement park in Orange County, California, which holds a popular “Knott’s Scary Farm” event each year.

If I manually change my location to tell Google that I’m in Des Moines, Iowa, that particular suggestion goes away and is replaced by “coupons for best buy.”

Similarly, if I go to Google UK, I get suggestions like:

• coupons uk
• coupons and vouchers
• coupons for tesco

Tesco is a major UK supermarket chain, just one reflection of how localized those suggestions are.

This is also why something like the Google Instant Alphabet or The United States of Autocomplete (shown below) — while clever — aren’t accurate and never can be, unless you’re talking about the suggestions shown in a particular region.

In short, location is important. The country you’re in, the state or province, even the city, all can produce different suggestions.

Language also has an impact. Different suggestions will appear if you’ve told Google that you prefer to search in a particular language, or based on the language Google assumes you use, as determined by your browser’s settings.

Previously Searched Suggestions

Google’s suggestions may also contain things you’ve searched for before, if you make use of Google’s web history feature.

For example, when I search for “rollerblade,” my suggestions look like this:

• rollerblade parts
• Rollerblade 2009 Speedmachine 110
• rollerblades
• rollerblade wheels
• rollerblade

The first two come from my search history. That’s why they have the little “Remove” option next to them.

Personalized suggestion like these have been offered since May 2009. The only change with Google Instant was that they were made to look different, shown in purple similar to how links look at some web sites, to indicate if you’ve clicked on them before.

How Suggestions Are Ranked

How are the suggestions shown ranked? Are the more popular searches listed above others? No.

Popularity is a factor, but some less popular searches might be shown above more popular ones, if Google deems them more relevant, the company says. Personalized searches will always come before others.

Deduplicating & Spelling Corrections

There a small degree of deduplicating and spelling correction that happens in the final suggestions that show, Google says.

For example, if some people are typing in “LadyGaga” as a single word, all those searches still influence “Lady Gaga” being suggested — and suggested as two words.

Similarly, words that should have punctuation can get consolidated. Type “ben and je…” and it will be “ben and jerry’s” that gets suggested, even if many people leave off the apostrophe.

Freshness Matters

Google Autocomplete also has what the company calls a “freshness layer.” If there are terms that suddenly spike in popularity in the short term, these can appear as suggestions, even if they haven’t gained long-term popularity.

A good example of this was when actress Anna Paquin was married. “Anna Paquin wedding” started appearing as a suggestion just before her big day, Google says. That was useful to suggest, because many people were starting to search for that.

If Google had relied solely on long-term data, then the suggestion wouldn’t have made it. And today, it no longer appears, as it didn’t maintain long-term popularity (though “anna paquin married” has stuck).

How short-term is short-term? Google won’t get into specifics. But suggestions have been spotted appearing within hours after some search trend has taken off.

Why & How Suggestions Get Removed

As I said earlier, Google’s predictions have been offered for years, but when they were coupled with Google Instant, that sparked a renewed interest in what was suggested and what wasn’t. Were things being removed?

Yes, and for these specific reasons, Google says:

• Hate or violence related suggestions
• Personally identifiable information in suggestions
• Porn & adult-content related suggestions
• Legally mandated removals
• Piracy-related suggestions

Automated filters may be used to block any suggestion that’s against Google’s policies and guidelines from appearing, the company says. For example, the filters work to keep things that seem like phone numbers and social security numbers from showing up.

Since the filters aren’t perfect, some suggestions may get kicked over for a human review, Google says.

Hate Speech & Protected Groups

In terms of blocking hate and violence suggestions, it’s not that everything possibly hateful gets blocked as a suggestion.

For example, “i hate my mom” and “i hate my dad” are both suggestions that come up if you type in “i hate my.” Similarly, “hate gl” brings up both “hate glee” and “hate glenn beck.”

Instead, hate suggestions are removed if they are against a “protected” group. So what’s a protected group?

Google doesn’t actually define this on its Autocomplete help page. However, a Google AdWords help page has a rundown on what Google’s long-considered to be protected groups:

• race or ethnic origin
• color
• national origin
• religion
• disability
• sex
• age
• veteran status
• sexual orientation or gender identity

Even “majority” groups such as whites get covered by this, under the “color” category. That seems to be why “i hate white” doesn’t prompt a suggestion for “i hate whites,” just as “i hate black” doesn’t suggest “i hate blacks.”

However, in both cases, other hate references do get through (“i hate white girls” and “i hate black girls” both appear). This is where a human review may happen, if the reference is noticed.

Legal Cases & Removals

Google blocks some suggestions for legal reasons. For example, last year, Google lost two cases in France involving Google Autocomplete.

In the first, Google was ordered to remove the word “arnaque” — which means scam — from coming up as a suggestion for when someone typed in the name of a distance learning company.

Google appears to have done this, when I checked today. Google would not say if it is appealing the case or whether this applies to preventing the word “arnaque” from appearing next to any company’s name.

From some limited testing, I think Google is preventing from “arnaque” from appearing after any company name but not before (“arnaque paypay” and “arnaque groupon” are suggestions).

In the second French autocomplete case, a plantiff — whose conviction was on appeal — sued and won a symbolic 1 euro payment in damages over having the words “rapist” and “satanist” appearing next to his name.

The plantiff’s name wasn’t given in the case, so I can’t check that the terms were removed as ordered. Last year, Google said it would appeal the ruling. The company gave me no update on things when I asked for this article. It doesn’t seem likely that this has caused Google to drop having such terms appear next to the names of other people.

Yesterday, news broke about Google losing a case in Italy involving suggestions. Here, a man sued over having the Italian words for conman and fraud appearing next to his name.

I can’t check if Google has complied with the ruling, because the man’s name was never given — nor does his lawyer make clear if Google has complied. It’s also unclear if this ruling is causing such terms to be dropped in relation to anyone’s name (this seems unlikely).

I asked Google about this but was only given a standard statement:

We are disappointed with the decision from the Court of Milan.  We believe that Google should not be held liable for terms that appear in Autocomplete as these are predicted by computer algorithms based on searches from previous users, not by Google itself.  We are currently reviewing our options.

In the US, Google won a case last month waged by a woman unhappy that the words “levitra” and “cialis” appeared near her name. That case largely involved arguments about commercial infringement, rather than taking a libel stance.

Postscript: Sean Carlos of Antezeta has more on the Milan case here.

Controversial Cases

Aside from legal cases, Google’s suggestions have occasionally become news controversies. Typically, Google responds to these with a standard answer, which goes like this: the predictions are based on how people search, not by any particular “agenda” that the company is trying to push.

Google tells me it doesn’t typically comment more in these cases, because it doesn’t want to be in the position of having to issue a detailed response for any oddity that someone spots. Still, Google did open up about two examples of strange suggestions that have come up in the past.

One involved the suggestion “climategate,” which oddly disappeared shortly after appearing. My Climategate: Just How Popular Is It, According To Google? story from December 2009 has more about this.

Blame that aforementioned freshness layer, says Google. Back when this all happened, the freshness layer had a gap that allowed spiking queries to appear for a short period of time, then disappear unless they gained more long term popularity.

That gap has since been reduced. Spiking queries stay around longer, then drop unless they gain long-term traction. The “climategate” suggestion didn’t catch on and so disappeared. It wasn’t manually removed, as some assumed, Google said.

Interestingly, looking today, “climategate” still hasn’t gained enough long term popularity to come up as a suggestion at Google. But over at Bing — which, of course, uses its own unique suggestion system — it is offered.

In another case, a search for “islam is” was producing no suggestions while searches for other religions were — including negative ones. Our Islam Is … Blocked By ‘Bug’ In Google Suggest story from January 2010 has more about this.

As it turned out, there was a human error involved, Google told me.

Those suggestions had been escalated for human review as possibly being hate-related. A block was placed, because someone assumed that Islam as a religion met the protected group criteria.

But in fact, Google Autocomplete does not consider religions to be protected groups (I’ll get back to this). So other religions didn’t have a filter established for them.

Today, “islam is” brings back some negative suggestions, just as is the case with other religions.

Nationalities Briefly Protected; Religions Not

Feeling confused about who get protected, at this point? So am I.

Remember when I listed what a protected group was, according to Google, above? That included religions, but that’s the definition that Google AdWords uses, not Google Autocomplete.

Similarly, Google’s YouTube has its own definition of protected groups:

Protected groups include race or ethnic origin, religion, disability, gender, age, veteran status, and sexual orientation/gender identity.

National origin isn’t on that list. Indeed, it wasn’t on the unpublished list that Google Autocomplete uses until last May, when Google began to filter suggestions related to nationality. Search for “americans are,” for example, and you got nothing.

To me, it’s kind of crazy. Why protect nationalities but not religions? And why wouldn’t suggestions like “jews are cheap” or “jews are racist” be considered against a protected group, in terms of a race or ethnic group?

Google gave me this statement on the topic (the brackets aren’t me removing words but instead how Google indicates a search term):

Simply put, nationalities refer to individuals, religions do not. Our hate policy is designed to remove content aimed at specific groups of individuals. So [islamics are] and [jews are] or [whites are] would possibly be filtered, while queries such as [islam is] and [judaism is] would not because the suggestions are directed at other entities, not people.

Sorry, I’m not convinced by this. Worse, when I did some double-checking today, the previously established nationality filter — which the statement defends — appears to be turned off. Yes, Americans are again fat, lazy and ignorant, as Google’s “predictions” suggest, and the French are lazy cowards.

Can You Request Removals?

As you can imagine, some people would like to have negative suggestions removed. However, as explained, Google only does this in very specific instances. The company doesn’t even have a form to request this (though there is a help page on the topic, that suggests leaving comments in Google’s support forums).

Should businesses be allowed to request removal of suggestions? It’s not something that Google wants to arbitrate. Jonathan Effrat, a product manager at Google who works on Google Instant, told me:

Unfortunately, we won’t do a removal in those situations. A lot of times people are searching for it, and there’s a legitimate reason. I had a friend who used to work for a company, and the company name plus “sucks” was a suggestion, and that was the reality. It’s not really our place to say you shouldn’t be searching for that.

There are signs that Google has been pulling back by suggesting “scam” along with company names, but despite these reports, you can still find examples where this still happens. Google hasn’t commented if it’s actually made any change like this, by the way.

What About Piracy?

Of course, Google recently did decide that people shouldn’t be searching for things, in the case of online piracy, when it began blocking terms it deemed to be piracy-related in January.

That took out — and continues to take out — suggestions for some sites that may also be used for legitimate reasons. To be clear, suggestions were removed, not the sites themselves.

Want to read the Wikileaks files directly? BitTorrent or uTorrent have software that will allow you to do this. But today, Google won’t auto-suggest their names as you begin to type, deeming them too piracy related.

Aside from taking out some potentially innocent parties, the whole thing feels kind of hypocritical. Why does Google feel it needs to go over-and-above to protect searchers piracy-related suggestions when there are a range of other potential harmful ones out there?

The answer, in my view, is that this is a PR battle Google wants to win as studios and networks accuse it of supporting piracy and seek to enlist the aid of the US Congress. Dropping piracy suggestions is an easy gift, especially when Google’s not proactively removing the real issue, sites that host pirated content in its own results. It’s also a gift that might help it get network blocking of Google TV lifted.

And Fake Queries?

Meanwhile, another issue has gained fresh attention — the ability for people to “manufacture” suggestions. In particular, Amazon’s Mechanical Turk is a well-known venue where people can request that others do searches. When enough searches happen, then suggestions start appearing.

Brent Payne is probably one of the most notable examples of someone deliberately doing this “above the radar,” so to speak. He ran a series of experiments where he hired people on Mechanical Turk to do searches, which (until Google removed them) caused suggestions to appear:

Tempted to try it? Aside from potentially violating Mechanical Turk’s terms, Google says doing so is something it deems spam and will take corrective action against, if spotted.

What action? So far, that seems to be limited to removing the manufactured suggestions.

A Suggestion For Google’s Suggestions

As I said, Google Instant prompted renewed attention about Google’s suggestions — along with debate about whether Google should be offering suggestions at all, given the reputation nightmare they can bring to some companies and individuals, as well as offense they bring to other groups. On the flipside, there’s the usefulness of them.

Here’s a case that illustrates the balancing act. Last year, a skydiving company contacted me, concerned that searches for its name brought up a suggestion of its name plus the words “death” or “accident.” Yes, the company had someone who died in a jump.

That’s something harmful to the company, even if true. Skydiving is by its nature an extremely dangerous sport, and the suggestion gives no guidance about whether the company was somehow at fault. It just immediately suggests there’s something wrong with the company.

However, it’s also incredibly useful for searchers, as a way for them to refine their queries in ways they might not expect.

Still, I think the balancing act should tip back toward not offering up anything negative about any person, company or group. No nonsense about “protected groups.” Just kill the negative suggestions, period.

This is a suggestion for all the major search engines, by the way. Enough singling out Google, when these types of examples can be found easily on Bing and Yahoo, also.

If there are negative things that people want to discover about a person, company or group, those will come out in the search results themselves, and mixed in with more context overall — good, bad or perhaps indifferent.

Yes, many Americans know they’re stereotypically seen as fat. Other nationalities and religious groups also know that there are many hurtful stereotypes about them. But who wants Google seeming to tell them that?

Yes, Google’s correct in saying that the suggestions it shows reflect what many people are searching for — and thus think.

Still, parroting harmful thoughts “searched” by others doesn’t make those things any less hurtful or harmful. And by repeating these things, there’s an argument that search engines simply makes the situation worse.

Google had hoped to circumvent Chinese government control by redirecting Google.cn users to the uncensored Google.com.hk. But the Chinese have blocked that site on occasion as well. While some have speculated that Google will return to China proper, that’s not clear. The mixed signals from Google reflect an apparently heated debate on China internally within Google.

Recall that in January Google decided it would no longer comply with government censorship, in part because of alleged government-backed efforts to hack into Gmail. Since that time Google and China watchers have observed and speculated about how the Google-China drama would unfold. That’s still not clear, but it would be very difficult for Google to go back into China without losing face unless there were some concession by the Chinese government.

So far there has been no indication that the Chinese government will compromise or relax its censorship policies.

Here’s some of our past coverage on Google-China:

• Google No Longer Redirecting Google China To Google Hong Kong
• Google’s China Gambit: Day Two Reaction
• Google Just Says No To China: Ending Censorship, Due To Gmail Attack
• Will Google Leave China On April 10th Forever?
• Google To Give Google China More Freedom

When the New York Times was a pure newspaper, it was easy to appear agnostic about its editorial coverage, with no reason to play favorites with one business or another. But as the New York Times has branched out, making investments in external companies, it has acquired pecuniary [that means financial, by the way] incentives to favor those over rivals.

The New York Times argues that its behavior is kept in check by competitors like The Wall Street Journal or the Tribune Company. But the New York Times has become the default newspaper for many internet and print readers, with home delivery in over 340 markets and is, in fact, the nation’s largest seven day newspaper. Competitors are a click away, but a case is building for some sort of oversight of the gatekeeper of news.

In the past few years, the New York Times has come under accusations that it is too liberal or that the things it writes can’t be believed. It once employed a reporter by the name of Jayson Blair who resigned in May 2003, admitting that he just made things up. There have been other controversies involving the newspaper, enough to make a Wikipedia page of them. Mysteriously, that was deleted just a month ago.

These accusations and concerns may have merit. Or maybe they don’t. We can’t be bothered to look into the possible motivations of those making them, to investigate how true they might really be. Some of those accusing the New York Times of being unfair may not deserve coverage or may be seeing a liberal bias because of their own conservative ones.

Still, the potential impact of the New York Times algorithm on the internet economy, not to mention the US economy, the US government and the world as a whole is such that it is worth exploring ways to ensure that the editorial policy guiding the New York Times is solely intended to improve the quality of journalism and not to help other businesses that the New York Times owns or the bottom line of its for-profit owners.

Some early suggestions for how to accomplish this include having the New York Times explain with some specified level of detail the editorial policy that guides what it decides to covers, what it doesn’t decide to cover, why it chooses to write a particular headline with a particular angle, to show all versions of a newspaper story that is written from start to finish, to reveal what’s been edited out. Another would be to give some government commission the power to look at all these aspects, perhaps the power to reside within the newsroom and ensure fairness.

The New York Times provides an incredibly valuable service, and the government must be careful not to stifle its ability to innovate. Forcing it to publish the algorithm or method it uses to evaluate would allow any business or government agencies to game the rules in order to obtain positive coverage — destroying its value as a newspaper. Requiring each algorithm tweak to be approved by regulators could drastically slow down its improvement and regular reporting operations.

Forbidding the New York Times to branch out into new areas — such as when it invests in things like a job search engine (here) or a company behind a free blogging tool (here) — might reduce the diversity of its financial health, even if it does help ensure the company won’t favor these products over those produced by rival companies on the internet that it also covers.

With these caveats in mind, if the New York Times is to continue to be the main map to the news and information highway, it concerns us all that it leads us fairly to where we want to go.

And Now, Without The Satire…

For those who didn’t see it, the piece above is a satirical rewriting of yesterday’s New York Times editorial, The Google Algorithm, which suggested that Google needs to be regulated, since its an important “gateway” that faces recent accusations of bias.

I’ve been covering the search space closely for nearly 15 years, from before Google itself even existed, so I have seen these types of claims far longer and examined them in far more depth than what went into that New York Times editorial.

My guess is that the editorial staff (the staff that writes the newspaper’s editorials, which are opinion pieces, which is confusing when the newspaper also has an editorial staff that writes “editorial” stories elsewhere that are supposed to be unbiased) spent about an hour or so discussing recent Google news, then someone was probably assigned to write the editorial and invested all of about three hours on it.

That’s not much time or care for a major and well-respected newspaper (in many quarters) to decide the government should evaluate “fairness” when it comes to making editorial judgments in search results, be they from Google or any other search engine.

Editorial Independence

Search engines are very similar to newspapers. They have unpaid “organic” listings, where usually (though not always), a computer algorithm decides which pages should rank tops. The exact method isn’t important. What’s important is that those unpaid listings are the search engines’ editorial content, content it has solely decided should appear based on its editorial judgment.

Search engine also have paid listings, advertisements, which aren’t supposed to influence what happens on the editorial side of the house. We even have FTC guidelines ensuring proper labeling of ads and intended to protect against “advertorials” in search results.

It’s a church-and-state divide with good search engines, just as it is with good newspapers.

What the New York Times has suggested is that the government should oversee the editorial judgment of a search engine. Suffice to say, the editorial staff of the New York Times would scream bloody murder if anyone suggested government oversight of its own editorial process. First it would yell that it has no bias, so oversight is unnecessary. Next it would yell even more loudly that the First Amendment of the US Constitution protects it from such US government interference.

First Amendment Covers Search Engines

Guess what. The First Amendment protections of freedom of speech and freedom of the press apply to more than newspapers. In fact, they apply to search engines. The courts have said so, most clearly back in May 2003, in the SearchKing case:

PageRanks are opinions–opinions of the significance of particular Web sites as they correspond to a search query … accordingly, the court concludes Google’s PageRanks are entitled to full constitutional protection.

The Gatekeeper Argument

Ah, but that was 2003. Google wasn’t so powerful or deemed as a “gatekeeper” then. Now it controls everything!

Wrong. Even back then, Google was being paraded as being too big, too powerful (see 14 “Is Google Evil?” Tipping Points Since 2001). Heck, consider this newspaper article from December 2002:

Google has become enough of a Web gatekeeper that its leads now prop up plenty of commercial sites….

Google uses many variables in its automated ranking process, but a key one is the number of other pages linking to a particular page, because links can serve as a kind of endorsement. Google regularly shuffles its rankings to reflect changes in its own methods or in the Web’s link structure and content.

That makes its free listings a risky platform on which to build a business. Google also routinely slashes the rankings of sites that appear to be using tricks, like hiding keywords in invisible text that can be read only by search engines….

The free ride may not last, however. Ms. Johnson of Forrester says larger companies have been discovering the power of search engines and site optimization. As was the case on eBay when big retailers moved in, search listings are becoming less democratic. “It’s going to be more and more difficult for small sellers to get noticed,” she said. “The free listings lunch may be ending soon.”

That’s from the New York Times, by the way — reporting how almost ten years ago, Google was a “gatekeeper” where the “free lunch” for small businesses might be ending.

What’s happened since then? Small businesses, along with big businesses, have continued to do just fine. Google has handled hundreds of BILLIONS of searches over that time.

If Google was seriously abusing its “gatekeepter” status, you’d expect to hear billions of complaints about anti-competitive behavior in those billions of searches. Instead, the loudest voice we hear is from a tiny shopping search engine virtually no one has heard of, Foundem, screaming that we need “search neutrality” because Google’s tried to keep it out for competitive reasons.

That’s right. Google’s smart enough to build this supposed search monopoly, but then when it decide to whomp the competition, it’s Foundem that it finds a threat. Not Amazon. Not eBay. Not Yahoo. Not Bing. Foundem.

Please.

By the way, many people forget that Yahoo was once considered the web’s gatekeeper. There were even calls in the late 1990s that the government should step in since Yahoo’s editors could make or break a business depending on if it listed them or how it did so.

Somehow, businesses survived Yahoo. Somehow, businesses have survived Google’s supposed gatekeeping. A key part of that is down to the fact that business online do receive traffic outside of search engines, in particular with Facebook and Twitter sending some sites more traffic that Google.

Google’s Plenty Transparent

Still, shouldn’t Google share more about how it creates its algorithm? Compared to the New York Times, Google’s a model of transparency. Consider:

• Google will list EVERY site that applies for “coverage” unlike the New York Times, which regularly ignores potential stories
• If Google blocks a site for violating its guidelines, it alerts many of them. The New York Times alerts no one
• Google provides an entire Google Webmaster Central area with tools and tips to encourage people to show up better in Google; the New York Times offers nothing even remotely similar
• Google constantly speaks at search marketing and other events to answer questions about how they list sites and how to improve coverage; I’m pretty sure the New York Times devotes far less effort in this area
• Google is constantly giving interviews about its algorithm, such as this one in February, along with providing regular videos about its process (here’s one from April) or blogging about important changes, such as when site speed was introduced as a factor earlier this year.

Heck, back in June 2007, Google allowed New York Times reporter Saul Hansell into one of its search quality meetings, where some of the core foundations of the algorithms are discussed. Very few outsiders have ever attended one of these meetings. Hell, I’ve never attended one (Google’s joked with me that I’d understand to too much). Letting Hansell into this meeting would be the same as if the New York Times let Google CEO Eric Schmidt sit in one of their daily news budget meetings or better, a discussion between an editor and a report about the approach to a particular story.

Google’s Subject To Other Regulations

The question isn’t whether we need a new search “truth commission” any more than we need a newspaper “truth commission.” The question really is whether Google’s acting anti-competitively. We have plenty of anti-trust laws that already can be applied (and have been, as in the case of the proposed Google-Yahoo deal in 2008).

Exhibit A for some of the anti-competitive complaints is the resurgence in accusations that Google “favors itself” with specialized search products.

Google’s offered more than web search for a very long time. Image searches, for example, stretches back to 2001. It is a search company. It is supposed to offer search products. It makes no sense to expect those search products to be merely listing web pages. If people are doing shopping searches on Google, it should evolve its product to have a specialized shopping tool. That’s what its users want.

Sure, that might hurt other shopping sites out there. Or, it might not, if they offer a better shopping search than Google. But it’s a ridiculous argument that Google should somehow send every shopping query out to another shopping search engine.

Imagine if you did a web search for something, say “iphone,” and every link you got lead to Bing, Yahoo and other search engines, which in turn showed their results for iPhone. That’s crazy. You came to Google for answers, to be lead directly to sites with those answers, not to be sent to another search engine and forced to search again.

Where Google’s potentially anti-competitive is if no longer sends people AWAY from its site (or maybe not — no one complains that Yahoo recirculates much more back into its own properties). Its shopping search product sends people away to individual merchants, for free. To understand this more, I’d recommend watching my short talk below, The Search Platform: Friend Or Vampire?

That’s not to say Google’s without faults. It is. In particular, things like “Places” pages that recirculate back into Google or the fact that many of the external sites it lists carry its AdSense ads are causes for concern if not outright complaint.

But by and large, Google’s been a net positive actor, from where I measure things. It deserves better than a knee-jerk reactionary editorial from what’s supposed to be one of the leading newspapers of the world.

Postscript: Google is now also out today with its own take on the idea of “search neutrality” in an official blog post: Our op-ed: Regulating what is “best” in search?