Banned: Not Your Usual Suspects

We reported previously how the sites had gone missing, something Bing described as keeping with long-standing policies against “thin” content but which came out-of-the-blue to some site owners.

The banned sites included CyberMonday.com, which is run by Shop.org, the group that created the entire Cyber Monday concept.

Also banned were BlackFriday2011.com and CyberMonday2011.com, which provided content to Yahoo Deals. Yahoo is one of Bing’s strategic partners. The Yahoo site itself, ironically, wasn’t banned.

Several of these sites had top listings were not deemed to be spam by Google, which has been on a campaign to remove content not deemed to have enough substance to it. Indeed, some of them had top listings with Google.

Now, despite being deemed to thin, several of these sites have returned to Bing. CyberMonday.com is back, as is BlackFriday2011.com. CyberMonday2011.com remains banned.

Algorithm Now Deems Them No Longer Questionable

Why the change? Bing sent this statement:

Black Friday and Cyber Monday are notorious times for spammers, and during this time Bing’s spam classification algorithm picked up this spam pattern and heightened its criteria.

Bing took proactive action to protect our users by removing questionable domains.

In an effort to protect our users some questionable domains may have been demoted or removed that some may consider legitimate sites.

We have since revised our algorithm which has led to some previously blocked sites returning to the index.

Really An Algorithm Change?

A search engine’s algorithm, an automated computer process, is like a recipe that measures many different factors to determine what content should rank well, as well as what to ban (watch our video here about this).

This algorithm change at Bing didn’t seem to weigh many factors. Instead, the change simply seemed to be that Bing banned any site that had the words “black friday” or “cyber monday” in their domain name, rather than an analysis of the content itself, to decide if it was too “thin” to retain.

Bing, however, did reconfirm to us that an algorithm was involved.

Bing Denies Favoring Itself

Several of those dropped have wondered if it was all part of an attempt by Bing simply to drive more traffic to a special section of Bing Shopping that was ranking (and still does) in the first page of listing for searches at Bing for black friday and cyber monday, like this:

Bing denied the move was designed to help itself:

Bing did not replace algorithmic results with its own sites.

Since its inception Bing has offered verticalized content like shopping and travel if it helps address a user’s intent.

These Instant Answers are available for a variety of topics and scenarios, such as shopping, checking stock prices, or stats for athletes.

More Information

For more background on the removals, including how it might help Google with accusations that it has banned shopping sites simply to help itself, see our past articles below:

• Bing Bans Holiday Deals Sites, Including One By Group That Created Cyber Monday
• Holiday Deals Sites Confirm: Bing Dropped Them Just Before Black Friday & Cyber Monday

Bing’s Version Of The Google Panda Update

If “thin is bad” sounds familiar, that’s because Google kicked off this trend earlier this year. The Google Panda Update was a change in how Google ranked web pages, designed to penalize pages deemed to be content-light.

Google’s change didn’t remove web pages. It just meant that pages deemed thin had less chance of ranking well.

In contrast, Bing’s change is more dramatic. It has completely removed some sites deemed thin from its search engine.

CyberMonday.com: Loved By Google, Banned By Bing

Consider the case of CyberMonday.com. The site is run by Shop.org, the digital division of the National Retail Federation. Microsoft — which owns Bing — is one of the Shop.org member companies, along with many other major brands.

Shop.org coined the term Cyber Monday in November 2005, as even covered by BusinessWeek by then. The following year, Shop.org created the CyberMonday.com web site to go along with the campaign.

On Google, CyberMonday.com currently ranks in the top results for a search on cyber monday:

On Bing, it doesn’t rank in the top results. Noteworthy, Bing’s own special page to promote Black Friday and Cyber Monday deals available through Bing Shopping gets top billing:

Bing Kills Thin Sites, Doesn’t Just Bury Them

CyberMonday.com doesn’t rank because it has no pages listed within Bing at all:

CyberMonday.com isn’t the only site impacted this way. A number of Black Friday-oriented sites have also gone missing, including:

• blackfriday.info
• theblackfriday.com
• black-friday.net
• 2011blackfridayads.com
• blackfriday.com
• blackfriday2011.com

Bing: Dropping Thin Sites Isn’t New

Why were they dropped? Bing told me:

One of our main goals at Bing is to deliver quality results for our users. Consistent with our guidance to site owners, websites that seem to rely mostly on affiliate content or that offer only thin content often don’t deliver the value searchers are looking for and may be demoted or removed from our index. This is something we continually refine and look at closely throughout the year.

Is this new? Does it impact more than Black Friday and Cyber Monday sites? I was told:

It’s nothing new, and follows guidance we’ve given on our webmaster site. We don’t have any specifics to share.

But It Sure Feels New

Despite what Bing says, this sure feels new. Bing’s statement says this is all consistent with its guidelines, with the link it emailed as part of that statement supposedly pointing to that guidance. But I found nothing there that gave the impression this was a long-standing policy.

The link leads to the Bing Webmaster Center blog, not to its publisher guidelines. Going through the posts myself, looking for any news that thin content gets you banned, the best I could find was this post in August that said:

The worst approach leaves you to seem “thin” in your approach.  This essentially means you aren’t really providing content of value.  An example of this would be a website aggregating content from multiple sources on one page.  Such an approach amounts to little more than a links page related to the query entered.

Thin’s not good, clearly, but that article doesn’t say that thin pages will be banned.

Heading over to actual publisher guidance area, the Guidelines For Successful Indexing doesn’t list thin content as a reason a site might be removed from Bing. Neither does the page about requesting reinclusion.

Instead, I had to go over to another section at Bing designed for searchers — not for publishers — to find this definition of spam that seems to include thin pages as something that might be removed:

Some pages captured in our index turn out to be pages of little or no value to users and may also have characteristics that artificially manipulate the way search and advertising systems work in order to distort their relevance relative to pages that offer more relevant information. Some of these pages include only advertisements and/or links to other websites that contain mostly ads, and no or only superficial content relevant to the subject of the search. To improve the search experience for consumers and deliver more relevant content, we might remove such pages from the index altogether, or adjust our algorithms to prioritize more useful and relevant pages in result sets.

I’m not sure how long this has been there; it’s certainly not in an area publishers would easily spot. It doesn’t seem to guarantee that sites that are thin will be completely blown out of Bing, in the way that these sites have been.

And Feels Targeted To Holiday Deal Sites

Despite what Bing says, this change does seem new — and specifically aimed at sites featuring holiday shopping deal offers.

The person who tipped us to these missing sites last week certainly thought this was a recent change (with the Thanksgiving break, it took until now to get a statement back from Bing).

Over at WebmasterWorld, Rob Woods noted the same in a discussion:

Bing just nuked all but one of the top black friday ad sites from the index. Not just penalized but completely removed all the top ranking domains entirely (except one).

Bing also just nuked any site with “Cyber Monday” in the URL including the .com which is the official site of the National Retail Federation which created Cyber Monday in the first place. Having launched their own black friday / cyber monday affiliate site it looks like they are trying to eliminate any competition from the SERPs. Virtually any site with either keyword in the domain has been completely removed from Bing’s index.

And Is The Opposite Of What Bing Said About Panda

Another oddity is that earlier this year, when asked about Google’s Panda effort, Bing suggested that it wasn’t relevant for Google to be applying sitewide penalties. In an interview with Eric Enge, Bing Director Stefan Weitz said:

Google’s Panda Update was an interesting event. I saw reports recently on DemandMedia showing they were down 40% on their traffic. What this speaks to is the necessity to look at page level quality. I think one of the things that started the work on Panda was the JC Penney paid link issue which called into question the quality of PageRank.

Google initially responded by blocking the entire JC Penney domain for a few days. We thought that hurt the users because we did the same thing in a test. We blocked all JC Penney internally and asked our human ranking systems “does this result for the search phrase “comforters” look better or worse after this change?” Everyone said it looked worse because they expected to see JC Penney there.

What it told us was there are different ways to classify quality of pages. We have page level classifiers that look at every page we index that attempt to discern a quality score. It looks at things like reading levels, number of ads versus content, length of words, length of page, all those standard things, and some not so standard things as well.

In short, Bing believes it’s important that each page be individually assessed for its relevancy and further says it has the technology to do this. However, in the case of these sites, none of this is happening. They were just completely wiped out.

Are These Sites That Bad?

Of course, I’m not suggesting that Bing should keep crappy sites in its search engine. It’s just pretty strange to discover that the site run by the group that started the whole Cyber Monday idea was among those to get the axe, a group supported by Microsoft itself, a site that includes the Microsoft Store among the listings.

Sure, the Blekko search engine that’s been campaigning against thin content doesn’t list any pages from it, either. But Google – which also has guidelines against thin content — deems it worthy of a top ranking.

Similarly, these three sites are all deemed good enough the top results for black friday on Google:

The same sites are completely banned by Bing. They have no pages at all listed, and so they certainly don’t rank for black friday:

Those two arrows? One points to the aforementioned page run by Bing itself offering Cyber Monday and Black Friday specials through Bing Shopping.

The other one, higher up, highlights how BFAds.net managed to survive the culling, even though I’m hard pressed to see the difference between this post on that site versus what Cyber Monday’s home page offered.

Can you spot the difference?

BFAds.net is on the left; CyberMonday.com is on the right.

Thin Is Bad; Thinner Than Thin For Yahoo Is OK?

In another inconsistency, consider this search for black friday ads at Bing:

The arrow points to a page at Yahoo Deals with Black Friday ads:

See those ads that the arrow points to? Those come from BlackFriday2011.com. Click on any of them, and you go to a page at that site — which itself is banned by Bing.

Now that’s irony. BlackFriday2011.com is deemed to thin for Bing, but a page that points to all that thin content? That’s apparently thick enough to keep.

Blowback On The Antitrust Front

Google is one company that will getting a nice holiday bump from the removals. Google has been under fire that it manipulates its results to promoting its own content and keep competitors out.

Microsoft has been one of the companies funding efforts to fight Google on this front, such as through FairSearch. But now, Bing appears to be doing exactly what Google’s accused of. Sites have been dropped that potentially compete with the special Bing Shopping page that’s ranking well.

Like Google, Bing has a dedicated shopping search engine. Like Google, searchers are automatically suggested to use this for relevant queries, such as this unit that’s inserted into a search for digital cameras, showing shopping results:

In contrast, this unit for the special Bing holiday shopping page isn’t showing shopping results:

Instead, it’s designed to drive people to a particular page in the shopping area. It also carries a special tracking code. Both things are unusual, red flags that this is something being done to promote Bing above and beyond what its regular relevancy algorithms would do.

As for the removals, Foundem — another member of FairSearch — has lead the charges that it and others have been banned from Google not for quality issues but instead for anti-competitive reasons.

Google’s Panda Update, Foundem recently argued in a white paper (PDF), was just a continuation of this process:

Panda marks a significant escalation in Google’s undeclared war on its vertical-search rivals. So far, few have made a connection between Panda and the various antitrust Investigations into Google. But Panda isn’t just relevant to these investigations; it is central to them.  Despite being widely touted as an attack on content-farms—which are almost the polar opposite of vertical search services—Panda also marks an aggressive escalation of Google’s vertical-search targeted, “lack of original content” penalties….

With Panda, Google is now targeting many established vertical search brands, as well as emerging ones. Still mindful that it cannot openly penalise well-known competitors, Panda’s algorithmic demotions are more subtle than their predecessors: although affected sites do not completely disappear from Google’s search results, they are systematically demoted to a point beyond the reach of most users, and so receive little or no traffic from this vital channel.

If Google’s Panda Update is bad for merely demoting sites for lacking original content, it’s hard to see how Bing completely removing sites for exactly the same reason is going to help those arguing this aspect of the antitrust investigations against Google.

Postscript: I’m waiting to hear back from Shop.org. They’re checking on the situation, having heard about it only today, apparently after this article appeared. My guess is that they didn’t even notice they were dropped from Bing, probably because Google sends so much traffic to them that not being in Bing didn’t register.

Postscript 2: See our follow-up post at Marketing Land, Holiday Deals Sites Confirm: Bing Dropped Them Just Before Black Friday & Cyber Monday.

Postscript 3: See our further follow-up story, Banned Holiday Deal Sites Return To Bing.

Now, according to PaidContent, a proposed amendment to a pending UK law (“Digital Economy Bill“) would permit search engines to index any or all of the content on a “publicly accessible website” through a “presumed . . . standing and non-exclusive license.” Accordingly, indexing of third party content, however extensive, would not be liable for copyright infringement. This amendment (whose adoption is uncertain) is from self-described Libertarian-Conservative Lord Lucas. But it’s a fairly radical provision.

It’s unclear how using robots.txt would play here in terms of burdening the rights conferred by the amendment, which are much broader than “fair use” in US copyright law.

If the law and the Lucas amendment were to be adopted, imagine a scenario where a raft of new “search engines” appeared in the UK and simply cloned large portions of “publicly accessible” content sites or copied articles wholesale. This might create havoc for publishers, legitimate search engines and end users. It’s unlikely to come to that but such scenarios are clearly implicated by the wide-ranging immunity of the amendment.

ACAP started development in 2006 and formally launched a year later with version 1.0 (see ACAP Launches, Robots.txt 2.0 For Blocking Search Engines?). This year, in October, ACAP 1.1 was released and has been installed by over 1,250 publishers worldwide, says the organization, which is backed by the European Publishers Council, the World Association of Newspapers and the International Publishers Association.

If that sounds pretty impressive, hang on. I’ll provide a reality check in a moment. But first, let’s pump ACAP up a bit more. Remember back in July, when the Hamburg Declaration was signed by about 150 European publishers? The short declaration basically said that intellectual property protection needs to be increased on the internet, in order to protect high-quality journalism.

ACAP: Save Our Content!

Enter ACAP, as a lynchpin to achieving the Hamburg Declaration’s dream. From the official release put out by the European Publishers Council, which organized the declaration:

We need search engines to recognize ACAP as a step towards acknowledging that content providers have the right to decide what happens to their content and on what terms. The European Commission and other legislators call on our industry constantly to come up with solutions – here we have one and we call upon the regulators to back it up.

That quote is from Gavin O’Reilly, president of the World Association of Newspapers and News Publishers, Group CEO of Independent News & Media and chairman of ACAP.

To me, it reads like it’s the Wild West out on the internet. That search engines are doing whatever they want with content, with publishers having no control over what happens. ACAP would bring rules to the search engines, and those rules would have the force of law if some governmental bodies would force them on the search engines.

The Wild West Is Actually Tame

The reality is that the search engines do follow rules, ones they’ve created and enhanced over the past 15 years based on feedback from the entire web community (rather than from a select group of largely disgruntled news publishers). Moreover, in all this time there have been relatively few lawsuits over how search engines interact with news content. Only one really stands out in my mind, the case won by Belgian newspapers over being included in Google News.

It was an unnecessary lawsuit. The papers could have stayed out of Google News using existing controls. In fact, despite “winning” the lawsuit, the papers eventually sought reinclusion in Google News using existing standards (see Belgian Papers Back In Google; Begin Using Standards For Blocking).

Meet REP: On The Beat For 15 Years

What are those existing standards? Collectively, they’re called the “Robots Exclusion Protocol” or REP for short. REP is made up of:

• Robots.txt: created in 1994 as a way to block content on a server-wide basis using a single file (the robots.txt file)
• Meta Robots Tag: created in 1996 as a system to block on a page-by-page basis (see Meta Robots Tag 101: Blocking Spiders, Cached Pages & More for more about it)

The two standards both live at robotstxt.org, but they’ve never been updated there, nor is there any type of official group or organization behind REP. Instead, search engines have either unilaterally or collectively expanded what REP can do over the years. They serve as the de facto bosses of REP, Google in particular. If Google makes a change, other search engines often mimic it.

I used “robots.txt” in the headline of this article mainly because that’s often used by those who live and breathe this stuff as a common name for both parts of REP. But I’ll be sticking with REP for the rest of this article.

Some ACAP In Action

Enough of the preamble and background. Let’s roll our sleeves up and see how the two system compare, starting with something easy. How can you block ALL your pages from ALL search engines using REP? You’d make a two line robots.txt file like this:

User-agent: *
Disallow: /

How would you do it in ACAP? Again, just two lines:

ACAP-crawler: *
ACAP-disallow-crawl: /

Sounds easy enough to use ACAP, right? Well, no. ACAP, in its quest to provide as much granularity to publishers as possible, offers what I found to be a dizzying array of choices. REP explains its parts on two pages. ACAP’s implementation guide alone (I’ll get to links on this later on) is 37 pages long.

But all that granularity is what publishers need to reassert control, right? Time for that reality check. Remember those 1,250 publishers? Google News has something like over 20,000 news publishers that it lists, so relatively few are using ACAP. ACAP also positions itself as (I’ve bolded some key parts):

an open industry standard to enable the providers of all types of content (including, but not limited to, publishers) to communicate permissions information (relating to access to and use of that content) in a form that can be readily recognized and interpreted by a search engine (or any other intermediary or aggregation service), so that the operator of the service is enabled systematically to comply with the individual publisher’s policies.

Well, anyone with a web site is a publisher, and there are millions of web sites out there. Hundreds of millions, probably. Virtually no publishers use ACAP.

Even ACAP Backers Don’t Use ACAP Options

Of course, there’s no incentive to use ACAP. After all, none of the major search engines support it, so why would most of these people do so. OK, then let’s look at some people with a real incentive to show the control that ACAP offers. Even if they don’t yet have that control, they can still use ACAP now to outline what they want to do.

Let’s start with the ACAP file for the Irish Independent. Don’t worry if you don’t understand it, just skim, and I’ll explain:

##ACAP version=1.0

# Allow all
User-agent: *
Disallow: /search/
Disallow: /*.ece$
Disallow: /*startindex=
Disallow: /*from=*
Disallow: /*service=Print
Disallow: /*action=Email
Disallow: /*comment_form
Disallow: /*r=RSS

Sitemap: http://www.independent.ie/sitemap.xml.gz

# Changes in Trunk

ACAP-crawler: *
ACAP-disallow-crawl: /search/
ACAP-disallow-crawl: /*.ece$
ACAP-disallow-crawl: /*startindex=
ACAP-disallow-crawl: /*from=*
ACAP-disallow-crawl: /*service=Print
ACAP-disallow-crawl: /*action=Email
ACAP-disallow-crawl: /*comment_form
ACAP-disallow-crawl: /*r=RSS

OK, see that top part? Those are actually commands using the robots.txt syntax. They exist because if a search engine doesn’t understand ACAP, the robots.txt commands serve as backup. Basically those lines tell all search engines not to index various things on the site, such as print-only pages.

Now the second part? This is where ACAP gets to shine. It’s where the Irish Independent — which is part of the media group run by ACAP president Gavin O’Reilly — gets to express what they wish search engines would do, if they’d only recognize all the new powers that ACAP provides. And what do they do? EXACTLY the same blocking that they do using robots.txt.

So much for demonstrating the potential power of ACAP.

Well, how about the Wall Street Journal, backed by Rupert Murdoch, who’s been on an anti-Google bend of late. Same situation — the WSJ’s ACAP file is doing nothing more than what the robots.txt commands show. Actually, it does less. At least the robots.txt system allows for discovery of a sitemap file (more on this below).

How about the Denver Post? It doesn’t have an ACAP file, just plain old regular robots.txt file. Why’s that signficant? The CEO of the media company that owns the Denver Post — Dean Singleton — recently suggested he’d pull some of his content out of Google (see Hold On: Are More Papers Really Joining Murdoch’s Google Block Party?).

Singleton is also chairman of the Associated Press, which has been very anti-Google of late and which also is a backer of ACAP. So if ACAP allows the expression of control that publishers somehow don’t currently have, I’d expect the Denver Post to be among the poster children along side the Irish Independent and the Wall Street Journal.

Well, how about the Troy Daily News, which is one of the organizations that ACAP proudly lists as using its system. What’s happening with a rank-and-file publisher. From its ACAP file:

User-agent: *
Disallow: /private.asp
Disallow: /SiteImages/
Crawl-delay: 10
Request-rate: 1/10 # maximum rate is one page every 10 seconds
Visit-time: 0500-0845 # (GMT) only visit between 1:00 AM and 3:45 AM EST

#------------------------------------
##ACAP version=1.0
ACAP-crawler: *
ACAP-disallow-crawl: /private.asp
ACAP-disallow-crawl: /SiteImages/
#-----------------------------------------

Again, ACAP isn’t being used to express anything more than what’s already indicated in the robot.txt commands (the first section). Again, robots.txt actually goes beyond, as there’s support for a “crawl-delay” directive that ACAP doesn’t have.

That “request-rate” and “visit-time” telling search engines only to come by in the early morning hours? Have a chuckle at that. None of the major search engines recognize those commands. Similarly, visit the Hilton.com robots.txt file where you’ll see a similar but totally unrecognized command: “Do not visit Hilton.com during the day!”

Side-By-Side, REP & ACAP

OK, so even though no one’s using the special ACAP controls, let’s at least look at some of the key features and see how special they supposedly are. The table below lays out what REP offers against with ACAP.

In parentheses, I’ve noted the key commands used in both systems, for the technically inclined. Links lead to more information, as appropriate. Further below the chart, I’ve added more explanations as necessary.

Because REP has been extended by the major search engines, I’ve counted some areas as “Yes” for support if at least Google provides an option (given it has the largest marketshare of all). I’ve also noted the situation with Bing. As Yahoo search technology is slated to be acquired by Bing, I didn’t itemize its control offerings, since these will

For specific technical details on ACAP, see technical documents here. The easiest to comprehend is the implementation guide of Oct. 13, 2009. Also see the two crawler communication parts, if you want to dive in further.

OK, that’s the big chart. As you can see, there are some things that both systems provide and some things that are unique to each one. Here’s my personal take on the differences:

Jeers To ACAP!

Block Indexing: ACAP makes a weird distinction between blocking crawling (a search engine literally going from page to page automatically) and indexing (a search engine making a copy of the page, so that it can be added to a searchable index). For the major search engines, crawling and indexing are one and the same. I struggle to see an advantage to separating these out.

Cloaking: Those savvy to search engines know that Google hates cloaking, which is the act of showing a search engine something different than a human being would see. It’s often associated with spam. There are plenty of cases where people have shown misleading content to a search engine, in hopes of getting a good ranking. One example is from 1999, when the FTC took action against a site that was cloaking content that ranked for “innocent” searches like Oklahoma tornadoes and instead directed them to porn sites. The idea of a publisher forcing a search engine to allow cloaking would be somewhat similar to a newspaper being forced to write whatever a subject demanded be written about them.

Exact Snippet To Be Used: Similar to cloaking, allowing site owners to say whatever they want about a page sounds great if you’re an honest site owner. When you’re a search engine that knows how people will mislead, it’s not so appealing. In addition, sometimes it’s helpful to create a description that shows what someone searches for in context — and that doesn’t always happen with a publisher-defined description.

Annotation Blocking: It’s hard to interpret how this would work. Is Google’s SideWiki an annotation system, where comments are left alongside a publisher’s content but in a separate window? Or does this mean annotations on Google itself, such as SearchWiki allows? And should publishers be allowed to block people commenting about their pages on other sites? Does that block places like Yelp from reviewing businesses, if they link to them? Is Digg a ratings service? This option is a minefield.

Kudos To ACAP!

Maximum Snippet Length: Search engines are quoting more and more material from pages these days, it seems. The ability to limit how much they can use seems like a good idea that should be considered.

Meta Tag Only Commands: A number of controls such as blocking caching or snippets can’t be done in a single file. Now, for those using CMS systems, including free ones like WordPress, it’s relatively easy to add these codes to each and every page. But it would be nice to see the search engines add file-wide support for some of these options in the way that ACAP does.

Prevent Framing: I hate framing. I’d love to see a way to tell automated tools like URL shorteners that they can’t frame. But with the search engines, framing is pretty limited. Google does it with images, and you can block images from being indexed period, which eliminates framing. It does the same with cached pages, and you can block caching. Plus, it’s fairly easy for a site owner to block any type of framing.

Urgent Removal: If you’re a site owner, a system to get pages out of an index in a guaranteed period of time would be very convenient. However, this is probably better handled through webmaster tools that the search engines offer, as they allow a site owner to proactively trigger a removal, rather than waiting for visit from a crawler, which could take days. Ironically, at Google, they had a system to remove pages quickly. I wrote about it two years ago (see Google Releases Improved Content Removal Tools). But the documentation today is terrible. Little is explained if you’re not logged in. If you are logged in, the link for the webmaster version doesn’t work. The entire feature Google described in 2007 is gone.

Block Specific Parts Of Page: Who wants all their navigation being indexed, along with all the other crud pages often have on them? ACAP allows for only parts of a page to be indexed. Yahoo already offers this. Why not the others?

Hmm…

Permitted Places: The idea here is that you could allow your story to be listed in Google UK but not Google France, if you wanted. It might not be a bad idea, though that’s not usually the demand I tend to hear. Instead, site owners often are trying to figure out how to associate their sites with a particular country (Google has a tool for this).

Time Limit: You can time restrict when a page should be removed, a cached copy should be removed and more with ACAP. Google has some support here, though few use it, the search engine tells me. It also seems unnecessary. It seems far more efficient for a site owner to simply remove their own content from the web or block spidering, when ready. In either case, that causes it to drop from a search engine.

Jeers To Search Engines!

I think the biggest frustration in compiling this article was knowing that search engines do offer much control to publishers but finding the right documentation is hard. At Google, you can block translation, but it was difficult to find this page in the help pages offered to site owners. Bing has a way to block previews, but I couldn’t locate this within its help center. Google has a blog post saying that bing supports the nosnippets tag. Over at Bing, I couldn’t find this documented. FYI, Jane & Robot has a good guide that can help with those trying to understand all that’s allowed.

Cheers To Search Engines!

ACAP has focused on a publisher wishlist of options that often can be done in other ways. Don’t want thumbnail images in a search engine? OK, we’ll make a command, even though just blocking images would solve that problem.

In contrast, the search engine have added feature that have come from the outcries of many diverse site owners. Sitemaps, to provide a list of URLs for indexing. Crawl delay support. Richer snippets. Duplicate content tools, such as the canonical tag or parameter consolidation. They deserve far more credit than some news publishers give to them.

The Missing Part: Licensing

Did you catch the biggest option that ACAP does NOT provide? There’s no licensing support.

Remember how ACAP’s O’Reilly talked about how ACAP was needed to ensure “that content providers have the right to decide what happens to their content and on what terms.” ACAP really doesn’t provide that much more control than what’s out there now. It doesn’t give publishers significantly more “rights.” I mean, how many more “rights” can you have when you’ve got the nuclear option of fully withdrawing from a search engine at any time?

It’s the part I bolded that’s key, the “what terms” portion. ACAP is supposed to somehow support new business models. Part of the idea is that you might want to license your headlines to one search engine, your thumbnails to another, and this would all be bundled up in some partnership deal. To quote from the ACAP FAQ:

Business models are changing, and publishers need a protocol to express permissions of access and use that is flexible and extensible as new business models arise. ACAP will be entirely agnostic with respect to business models, but will ensure that revenues can be distributed appropriately. ACAP presents a win win for the whole online publishing community with the promise of more high quality content and more innovation and investment in the online publishing sector. ACAP is for the large as well as the small and even the individuals. It will benefit all content providers whether they are working alone or through publishers. A future without publishers willing and able to invest in high quality content and get a return on that investment is a future without high-quality content on the net.

Nothing in the ACAP specs I’ve gone through provide any type of revenue distribution mechanism, much less some type of automated handshake between a publisher and a search engine to verify permissions.

If REP & ACAP Files Could Talk

To illustrate this better, here’s a “real world” conversation of how ACAP supposedly works. I shared this recently with others on the Read 2.0 mailing list that I’m part of during a discussion, playing off some other conversation scenes that John Mark Ockerbloom had started. Several people said they found it helpful. Perhaps you will, too.

GOOGLE: Hi! I’m Google. Can you tell me if I can crawl your site?

PUBLISHER: Sure, but I might have some restrictions over what you can do.

GOOGLE: That’s cool. Just use a meta robots tag on particular pages to give me specific commands.

PUBLISHER: Well, on this page, I don’t want you to show a cached copy.

GOOGLE: Awesome, use the noarchive command. Done. What’s next?

PUBLISHER: On this page, you must always show the description I want shown for it.

GOOGLE: Use the meta description tag. We’ll consider that, but we can’t promise.

PUBLISHER: Dammit. You just want to rule the world.

GOOGLE: Look, we build description that are related to what someone searched for, dynamically. So if we find a page on your site, in response to a particular keyword, sometimes it makes sense to “snip” a description that contains that term from your page, so they immediately understand why your page is relevant to your search. And click to view it. That’s why we call them snippets.

PUBLISHER: Dammit. Do what I want. You’re not the boss of me.

GOOGLE: Well, we also get people who would say they have children’s games when instead, they have adult games — like porn. Seriously, true story. Plus, we’re the boss of us. I mean, is it OK if we declare that you must review us in the way we want in your publication.

PUBLISHER: Let’s move on. On this page, I don’t want any images to be used.

GOOGLE: Block them with robots.txt. Done.

PUBLISHER: This article, I only want you to list it for 30 days.

GOOGLE: Pull it down after 30 days. Or move the full article to a new location, and leave a summary page up, if you want remnant traffic. Or block it. Or use the unavailable-after meta tag.

PUBLISHER: I only want you to list this content if I have a paid partnership with you. My ACAP file will declare that to you.

GOOGLE: You have a paid partnership with us?

PUBLISHER: Well, not yet. But Murdoch’s promising us that will come.

GOOGLE: If you have a paid partnership with us, to give us permission to index your content, we know that internally. I mean, we don’t have many of those, and we’re not scanning the web and ACAP files to keep track of them. ACAP doesn’t even have a place for you to tell us this, anyway.

PUBLISHER: I don’t have a partnership. But I’m saying you should only index my content if you DO have a partnership. But you keep indexing it.

GOOGLE: Well, then block us. Surely you know if we don’t have a partnership or not. And you can use robots.txt to authorize indexing all you want.

PUBLISHER: But I want you to license our content!

GOOGLE: Yeah, we get that. Hey, check it out, have you see our free wifi at airports?

ACAP Not A Business Solution; Search Engines, Get Organized!

Overall, there are some ideas in ACAP that would be useful for the search engines to consider. However, there are many ideas outside of ACAP that would also be useful for them to consider. There’s nothing I see within ACAP that provides some type of crucial control that if only news publishers had, all their online woes would be over. What the news publishers really want are licensing agreements, and given that Google already has several of these without using ACAP (see Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers), I can’t see that having it somehow advances any business model changes.

Certainly the search engines need to get their act together more, however. It’s time to stop referring people to the REP site which is run by no one. It’s time to stop having a myriad of help pages scattered about within their respective sites. Yes, they should continue to have their own help pages (see Google’s webmaster help from here; Bing’s from here). But I’d like to see Google and Microsoft take the lead to also consolidate material into a common site, perhaps building off Sitemaps.org.

The controversy reached a peak this year when the NAR agreed with a local decision in Indianapolis that said real estate agents couldn’t let Google and other search engines index the property listings on their sites if those listings belonged to other brokers/agents. In March, the Indianapolis board sent a letter to some agents that essentially called search engines “scraper” sites:

“…the National Association of REALTORS® (NAR) is in agreement with our interpretation of the policy that the above described practice of ‘indexing your Web site’ as you have called it, is a method of scraping or reproducing the data”

At their meeting this week, the NAR Board of Directors revised its policy on home listings and search engines to say that participants “are not required to prevent indexing of their Web sites by recognized search engines.”

That Realtor.org link is also interesting for its comments about the development of REALTORS® Property Resource, which many say is essentially a national search engine for property listings.

The LA Times reported yesterday about a “Twitter hole” that it believed allowed Google special access to protected tweets, tweets made from Twitter accounts where owners have deliberately chosen not to have their tweets be made public.

Not so, said TechCrunch. The so-called protected tweets that the LA Times was finding in Google looked to be those made from before particular account holders locked down their accounts.

I checked with Twitter and got back the official word from their press office:

The TechCrunch article seems to sum up the confusion pretty well. It seems that the LA Times piece references tweets that were public but later the user protected the account, thus all subsequent tweets are private along with the profile. The tweets prior to that time cannot be un-cached.

Google has not been given a key to the castle…so to speak.

I’m good with this answer except for the word I’ve bolded — that formerly public tweets cannot be uncached. That’s incorrect.

Let’s take an example. Let’s assume you started your Twitter account in March. You started tweeting publicly, then in July decided to be private. Twitter doesn’t try to protect any of your past tweets. In fact, it’s pretty clear about this in its help page on the topic:

If you have a public account and you protect it, all updates after the time of protection will be protected. Your profile will only be visible to approved followers, and existing followers will not be affected.

Please note that tweets from protected profiles will not appear in search results. People will still be able to find your account using the Find People search tool but only people you’ve approved to follow your account will be able to see your tweets. Also note that any tweets posted while your profile is private will remain private indefinitely, and tweets posted while your account is public will remain public indefinitely

But Twitter could try to protect those formerly public tweets. As best I can tell, if you lock down an account, Twitter does make ALL tweets (formerly public or not) inaccessible to everyone accept those the account holder has authorized to see them. That includes search engines like Google or a Twitter-specific search engine like Topsy.

Well, if Google can’t get in to tweets after an account has been protected, why does it show some? And why does Twitter say this will happen?

Google seems to rely on the last information for a tweet that it could see. So you tweeted something in March. Google sees the tweet and records it. If in August, you protect your account. Google tries to revisit your tweets as it does with any web page, to make sure it has fresh information. It can’t get to any of your tweets now.

The ones from August, it never saw them, since they were never public — so it doesn’t list them.

That tweet in March? It keeps showing the information from the last time it saw it. And apparently, it will keep doing this for weeks or months.

Google didn’t send me a comment about this (I did ask, and I might get one later today). But that’s just how I know Google works and can see it specifically working with some protected tweets I investigated today.

As for Topsy, they told me:

Topsy only displays tweets that were once public. The refresh button will make them vanish if the account is now private.

Back to Google. Eventually it should update its old copy of the tweet with what it currently shows to non-authorized visitors, a message that says “This person has protected their tweets” (you can see this for millions of people on Google now).

Twitter could speed that process along by explicitly blocking tweets from a protected account with a meta robots tag configured to remove the page from the index entirely and from cached copies being allowed (the NOINDEX, NOARCHIVE commands).

That wouldn’t guarantee that formerly public tweets are all taken private, of course. Once something’s put out on the public web, it’s very difficult to pull it back. But it could help and seems an easy enough change to do.

If you have a protected account, also keep in mind that those who follow you might retweet what you tweet to the world. If you’re that worried, make sure you pick your followers carefully and regularly keep them informed that you don’t want things retweeted. Otherwise, be prepared for your private tweets to leak out.

For more about search and tweets, see my What Is Real Time Search? Definitions & Players post which cover some ways to make use of Google and its search options feature to drill-down into tweets.

Goldman explained that the court rejected her suit on three different levels. She sued Yahoo for Lanham Act false endorsement and it was denied by the court, as expected. The reason it was rejected? Goldman sums it up as follows:

The court denies Various’ 230 defense because its association with the banner ad was unclear. Having dismissed the federal Lanham Act claims completely, the court then declines supplemental jurisdiction over the state law claims. The court also rejects Stayart’s guffaw-inducing request for sanctions against the defendants for having the temerity of moving to dismiss her complaint.

What happened yesterday was that the DOJ provided procedural notification to the court that the US is exploring the potential anti-trust dimensions of the Google settlement. The so-called “fairness hearing” to finalize the terms of the settlement will happen on October 7 and the US has until September 19 to “present its views in writing.” That’s not a lot of time.

What did the DOJ say in its two paragraph letter notifying the court of its formal investigation? Here’s the body of the letter:

The Sherman Anti-Trust Act is mentioned in the first paragraph of the government’s letter. The Sherman Act (and related subsequent legislation) is intended to promote competition and prevent consolidation of power and control over markets. For example, Microsoft was found in 2000 to have violated the Sherman Anti-Trust Act for too closely tying Internet Explorer to the Windows operating system.

One of the central questions the DOJ is exploring in its review of the Google Book Search settlement is whether Google gains exclusive rights or benefits to which other parties don’t have access. Some legal scholars have argued that Google does gain what amounts to a monopoly over so-called “orphan books,” books that are out of print but not yet in the public domain and where copyright owners are effectively impossible to locate.

Google has repeatedly said that the agreement is non-exclusive but has not discussed that contention in precise detail. Questions in the anti-trust investigation will include how much control Google has over the “corpus” (material scanned), over pricing of book sales and to what degree third parties can gain access to the material. (New York University is holding a conference on the Book Search Settlement for interested parties on October 8, a day after the fairness hearing.)

Unlike the Google-Yahoo paid search deal that Google walked away from to avoid a legal battle with the DOJ, the Book Search Settlement is different. Because of the underlying lawsuit Google cannot simply walk away here. Putting aside some of the less substantive issues (e.g., whether the attorneys fees are too high) as a basic matter there are essentially only two potential outcomes: approval of the existing settlement or modification of selected terms. If I had to bet I would bet that we will see some sort of modification further expanding third party access to the scanned material.

You can read the settlement itself or take a look at the public-facing Google Book Settlement website for rights holders and publishers. For additional context and background here’s our “library” of Google Book Search stories.

When someone searched for [Zwartepoorte] and [bankrupt], Google showed Miljoenhuizen.nl and the snippet in the Google search results for Miljoenhuizen.nl was:

Complete name: Zwartepoorte Specialiteit: BMW…This company has been declared bankrupt, it has been acquired by the motordealer I have worked for Boat Rialto…

What is the issue? Well, Zwartepoorte did not go bankrupt and was upset when the Google snippet showed they did. So Zwartepoorte sued Miljoenhuizen.nl, not Google for this error, and won. The site owner had to change their web page so Google would remove the snippet.

This is pretty crazy if you ask me. Google algorithmically makes the snippets. No where on the page did the site say the company went bankrupt. In any event, the court required the web site to modify the page so Google would remove the automated snippet.

“The Court argues that it might be true that the website had no control over the functioning of Google but suggests that these questions about the opacity of Google’s functioning should be addressed in a broader context… then concludes that defendant had its own responsibility,” says van Hoboken, a PhD candidate at the Institute for Information Law at the University of Amsterdam.

Hat tip to Blogstorm for spotting this case.

After threatening a lawsuit against Google several years ago, AP won its first licensing deal with Google in 2006. Google was at pains to stress this wasn’t a deal designed to gain the rights to merely list AP stories. It was supposed to cover more “new” and “extensive” uses of AP material.

Bull. Today, you can still search at Google and find that it fails to list just one “originating” AP story in many cases. Sure, Google hosts AP stories, but it wasn’t like Google set out trying for that goal. It was just as happy as pointing outward. The bottom line was that the deal was a nice wrapper to go around getting the AP off Google’s back.

Now the AP’s been on again that without the right deal, it’ll pull content from Google. Technically, how they’ll do that is absurd. Will the AP robots.txt out its entire site? And ensure that all of its members do the same for any AP story? More likely, it would fall back to the lawsuit front.

But what’s the right deal? What’s Google paying the AP now, and what happened in three years that this amount wasn’t enough? Since the terms were never disclosed, the public can’t judge.

And the public needs to judge, especially as newspapers have secret stealth meetings (which involve AP CEO Tom Curley) which involve sessions like:

Journalism Online: Presentation on proposed service to charge for access to newspaper content and to license that content that (sic) online aggregators

(For more on the stealth meeting, also see the Nieman Journalism Lab report and Techmeme).

See, some in the newspaper industry persist in the assumption that merely listing headlines and summarizing stories is a copyright offense. The AP itself wants to charge for this and already has guidelines that make some people think they might be violating fair use laws when they aren’t (see Do Newspapers Owe Google “Fair Share” Fees For Researching Stories?).

In other quarters, we have people like Forbes.com CEO Jim Spanfeller pulling $60 million dollar figures out of the air, about what he thinks Google owes him and suggesting that Google is helping to destroy “one of the core building blocks of our democracy.”

Since so much is at stake here — democracy itself! — it doesn’t seem right that Google and the AP will reach a deal that no one gets the details too. Shouldn’t all journalistic enterprises, be they traditional or not, understand exactly the value Google’s willing to pay to one with a big mouth and a staff of lawyers?

Moreover, shouldn’t Google be taking a stand on behalf of the majority of publishers who I’d argue have no problem with aggregators or search engines listing their content. A small number of publishers with inflated opinions about their importance are pushing for (and winning) concessions to compensate for their dying business models and lack of foresight. Aren’t the wrong people being rewarded?

You have to appreciate the rich irony. I can’t get the AP to talk with me at all (their execs are all busy, I’ve been told). When their CEO attends a stealth meeting, it’s not even the AP that first reports about it despite the AP being so convinced that they originate stories. Instead, the AP story points to an Atlantic blog post that broke the news. And Google, which touts being open any time it is convenient, is being hush-mouthed here.

Google needs to step up. Defend the “fair use” rights it believes it has on its own behalf and those of the greater web ecosystem. Alternatively, negotiate a deal to solve its AP problem but do it openly, so everyone else understands what exactly is being given up.