The study surveyed more than 2,200 online news consumers between December 28, 2009, and January 19, 2010, and found that more than half of online news readers use a major news portal on a typical day, and among younger news consumers (aged 18-29), more than two-thirds visit news portals like Yahoo News or Google News. The news portals beat out all other online news sources among all three age groups.

Also noteworthy is that the Facebook-related news sources scored dramatically higher than Twitter.

The study also cites Nielsen and Hitwise data that shows Yahoo News was the top online news site in 2009.

Some other interesting findings from the Pew study:

• About 71% of internet users, or 53% of all American adults, get their news online; that number has been relatively steady in recent years.
• Only 35% of online news consumers have a favorite site. Most news consumers rely on multiple sources to get their news.
• Of that 35%, only 19% said they’d be willing to pay for news online.

If you’re interested in news, and online news specifically, there’s a lot of thought-provoking material in the report. You can read it online or download the report via PewInternet.org or StateoftheMedia.org.

Now the Wall Street Journal reports:

New articles from the Associated Press have quietly started rolling out on Google’s news site in the past hour, ending a nearly seven-week absence stemming from contentious negotiations between the two parties . . .

Around Christmas, Google stopped adding new articles hosted on its news site in preparation for no new deal being reached when the contract was set to expire late last month. The two sides apparently agreed to continue negotiations as that deadline passed with no word either way.

Here’s Google’s official comment on the matter:

We have a licensing agreement with the Associated Press that permits us to host its content on Google properties such as Google News. The licensing agreement is the subject of ongoing discussion so we won’t be commenting further at this time.

Yahoo previously reached a deal to continue hosting AP content in Yahoo News.

See also: Where’s AP In Google News? Apparently In Limbo, As Contract Running Out

Cuban’s speech at the OnMedia conference was reported by several publications. I’ll quote from AdWeek:

“Google is a vampire, and you run scared,” he said. “There is no reason to be indexed in Google.”

And:

He said readers who find headlines via Google rarely convert to traffic, and publishers have a hard time monetizing that traffic.

“You haven’t gotten anything back except that you’ve turned into zombies,” Cuban said.

I’ve bolded the key parts that I’ll contrast against Cuban’s backing of Mahalo, which is supposed to be a human-powered search engine. Both Mike Arrington at TechCrunch and Mathew Ingram at GigaOM have counterpoints to Cuban’s speech you may also wish to read.

Mahalo: A Mark Cuban Investment

From the Mahalo press kit, which you can find via the Mahalo advertising page, we learn that Cuban is an investor in Mahalo:

Mahalo: It Loves Google, Even If Cuban Doesn’t

The press kit also teaches us other interesting things. We discover that Mahalo provides “high SEO value,” as you see stated here:

For those new to the acronym, SEO stands for search engine optimization, the practice of generating traffic from the free listings that search engines provide. As Google is the biggest of the search engines, anyone doing SEO is largely tapping into Google.

This means that Cuban is an investor in a company that does NOT see Google as a vampire. Quite the contrary. Mahalo sees Google as a valuable traffic resource. Indeed, another slide from the press kit has Mahalo bragging about how it ranks at the top of an actual Google search results page (as it still does when I looked today). We’re told how Mahalo pages “rank highly in search queries” as part of the slide:

Rather than inform advertisers that Google is a vampire to be blocked — Cuban’s advice, remember? — Mahalo touts the opposite, how it can “help our partners increase their search engine rankings,” as you see in this slide:

Indeed, Mahalo bills itself as quite the SEO shop, able to do keyword research and organic linking for clients — I mean advertisers:

So let’s go back to those parts I bolded from Cuban’s speech. No reason to be indexed in Google? You don’t get anything back? Mahalo — a Cuban investment — clearly believes the opposite.

Yes, Mahalo Does News

Ah, but Cuban was talking about news content! Those headline readers, they don’t convert for news sites. Things might be different for a non-news site like Mahalo.

Actually, Mahalo does quite a bit of news targeting. Let’s look at the current Mahalo home page:

I see news about the Oscar nominations, news about Justin Mentell’s death, Super Bowl 2010 news, news about the Grammy Awards. Lots of news. Mahalo — it’s a news site, among other things.

Look here, at the home page of the Wall Street Journal today, unquestionably a news site. I’ve pointed at four different news topics on it: Obama’s 2010 budget, the Toyota recall, the Haiti earthquake and the Apple iPad:

Mahalo has pages on all these topics:

Maybe Mahalo’s run by a bunch of idiots who are wasting their time going after these news topics to get traffic from Google. If so, Cuban — as a wise investor — should advise that Mahalo focus on things more likely to convert. Alternatively, Mahalo’s making money off these efforts and potentially, so might news sites with original content.

If Cuban Can’t Police His Own Aggregators…

Did I say original content? See, that’s another irony. This is also from Cuban’s speech:

Cuban dared newspapers to stop linking their stories to Google and to police other aggregators….

“Show some balls,” he said. “If you turn your neck to a vampire, they are [going to] bite. But at some point the vampires run out of people’s blood to suck.”

If that word “aggregator” is leaping out at you but doesn’t really mean much, check out my How Search Engines, Aggregators & Blogs Use News Content article. It explains what they are and how everything works. In short, an aggregator is a site that summarize content that lives on other sites.

Now let’s take a closer look at the Mahalo page about Justin Mentell’s death:

There’s a whole lot of aggregation going on there. If Cuban’s urging that aggregators be policed, perhaps he could start with his own investment – Mahalo. Add Icerocket to the list, too. That’s another Cuban-backed service which offers news search like Google does. Here’s Icerocket aggregating news headlines about Cuban’s own speech against search engines aggregating news content.

To really visualize how much Mahalo is aggregating, look at this illustration:

See that arrow? It’s pointing at the original content. It’s hard to find it on the page versus all the aggregated/scraped content that comes from other sources. The illustration, used with permission, comes from Aaron Wall’s excellent piece published last week, Mahalo SEO Spam Case Study. Be sure to read that article for a deeper look at some issues with Mahalo.

My, Mahalo, How You’ve Changed

Of course, not all Mahalo pages will be like the one above. Some of the news pages I mentioned earlier have short summaries about news along with some aggregated — but at least hand-selected — links.

Still, Mahalo seems a far cry from the human-crafted results it launched with. Consider this page about fires in Malibu, as it looked in 2008:

Today, it looks like this:

Most of of the curated links are gone. You get some fast facts, then a ton of the automated aggregation.

Conspiracy Time!

Now let’s put on our tin foil hats. Mahalo makes money tapping into breaking news topics. Cuban makes money off of Mahalo. Cuban advises media owners with original news content that they should drop out of Google, especially since those Google visitors are worthless. If they do drop out, who benefits from the spots that open up in Google? Potentially, Mahalo. Potentially, Cuban.

Nice advice.

Similarly, who else do we see as a Mahalo investor? Let’s go back to the Mahalo slides:

News Corporation. And what’s News Corp been saying about Google?

News Corp chairman and CEO Rupert Murdoch has questioned Google’s use of news content many times, saying it, along with other search engines and aggregators, just steal his content and engage in theft.

News Corp owns the Wall Street Journal, whose publisher Les Hinton called Google a vampire seven months ago, before Cuban seemingly borrowed that metaphor.

News Corp chief digital officer Jonathan Miller said last November that visitors from Google are the “least valuable” to him — another theme added to Cuban’s speech.

How Might News Corporation Gain?

Who benefits if media owners take News Corp’s advice and pull out of Google? Mahalo, potentially — which News Corp has a stake in. Mahalo, by the way, which was founded by Jason Calacanis who used to work for Miller, counts him as a mentor and perhaps was used to float a trial balloon about the Wall Street Journal considering an exclusive deal with Bing.

Alternatively, as I covered in Garlic For The Google Vampire last year, News Corp benefits if media owners bail out of Google because it, unlike most other publications, probably has “must carry” status. As I wrote:

The Wall Street Journal and the Associated Press carry enough weight — are arguably “must carry” publications — that they’ll probably land some Google cash, in the end. And Google won’t call it pay off money, when it happens. We’ll get a euphemism about the deals being done for “new” or “different” uses rather than the “right” to list links.

Other papers and organizations, which struggle much more than the WSJ or the AP, won’t get anything. That’s why I don’t think Google should do such deals secretly. More on that in my Open Letter To Google & The AP: Reveal The Licensing Terms article.

Back To Reality

Now take off the tin foil hat. I’ve painted a possible conspiracy theory. I don’t believe it. I don’t think News Corp has taken the stance it has in hopes of driving out all other media outlets. I actually think News Corp, like many news organizations, has some serious concerns about how news can thrive in a time when fair use means “anything goes” to some. I also think the company, like many other media organizations, reacts badly to legitimate fair use out there because it has yet to learn how to thrive in the digital world.

As for Cuban, I don’t think he made his speech as part of an uber-plan to help his Mahalo investment. I think he probably has some heartfelt views that content owners are somehow being ripped off. From November, my If Newspapers Were Stores, Would Visitors Be “Worthless” Then? post covers more about that, including comments back and forth between Cuban and I on the topic.

Enough Hypocrisy & Rhetoric

What I don’t like is the hypocrisy both News Corp and Cuban have shown on this issue. A Look At All The Sites Owned By Rupert Murdoch That ‘Steal’ Content from last November at Techdirt covers some of the aggregators that News Corp runs, despite its anti-aggregator stance. My article today illustrates Cuban’s backing of aggregation at Mahalo plus touches it at Icerocket.

The lines people want to draw on both sides of the fair use and aggregation debate aren’t that black and white. Nor does the rhetoric bring anyone closer to solutions.

Postscript: Mark Cuban has posted a response to my article here that I’d encourage people to read. A few comments from me about it.

The other day in New York I gave a speech at the AlwaysOn Conference …. when I speak to a group like this, rather than just shilling a product, service or position as many …. I try to put myself in the business shoes of the audience. Then I discuss what I would do if I owned, ran or invested in their business, and the approach I would take to some of the strategic issues of the day.

The concept of directing comments to a vertical segment of a market is nothing new. I have been doing it for more than 20 years. Yet for some reason, based on comments from a few folks over the past couple days, there are some relatively high profile people in the internet business that have a tough time grasping that concept.

I have no problem grasping that concept. There were parts of the speech where Mark made suggestions about what he would do that fit right into this. But he also made declarations about the value of search traffic that just frankly seemed wrong. That’s why he’s getting the reaction.

Look, let’s say Mark were speaking in front of an audience that didn’t believe in vaccinations as a way to prevent illness. If he then started declaring that vaccinations were unsafe, just to put himself in the shoes of those in the audience, plenty would have spoken out against it.

It’s one thing to speak to an audience’s concerns. It’s another thing to position yourself in believing and backing some of those same concerns. To date, Mark’s given every indication that he believes Google is a vampire that sucks publishers dry. He’s done that in some of his writings where he’s not speaking in front of any particular audience but, I assume, just speaking his mind to voice what he believes in a topic generally.

Further on, Mark writes:

I think he Danny  likes to banter to create traffic, smart on his part. But I also think he doesn’t fully understand all the business elements on some of the topics he has challenged me on.

Sure, I love traffic as much as the next person. But the most important thing to me is that important issues are covered, and that there is some balance where it needs to be.

I wrote my piece in response to Mark between midnight and 5am the day it was written. I simply couldn’t sleep after pondering his statements and having seen the Mahalo press kit recently. I wasn’t kept up because I thought “Oh, goody, I’ll get a lot of traffic out of this.” I was kept up because I love journalism, I know search, and I deeply hate when people spout off stuff that I find massively incorrect to media owners who desperately need better advice.

I don’t understand the business elements I’ve challenged Mark on? Well, for one thing, I came from a newspaper background. Five years working for daily newspapers. Not on the business side, nope. But that’s about five more years of newspaper experience than I believe Mark has.

I left papers because while I wasn’t on the business side, I could see clearly that they were going to fail when it came to the web. I watched my own paper at the time, the Orange County Register, struggle to decide if it should do AOL, Prodigy, CompuServe or something else. I jumped out in 1995 to do web development with a friend, instead, knowing it would be the web. Side note: the fact the OC Register has its domain name right now was due to our web development company. We registered it for them helped drag them a little bit online.

The web development company didn’t take off, so I struck out on my own as journalist. From literally nothing (and I do mean nothing, there was a time when I debated if I could by a London Underground transport card for an extra zone because of the extra $1 expense), I built a web site entirely around search. One that offered both free content and material behind a paywall, which helped it survive the dotcom downturn. A giant conference series along the way.

That all came from me seeing that search was a story, and that it was a story that deserved more than a once-a-year article about it that was commonplace for the magazines that I freelanced for in 1995. I thought we needed our own CNN of search news. So I started that businesss, a business that any media publisher could have easily have. And in 2006, the entire thing got sold for $43 million. From nothing. That’s not the $6 billion that Mark got from his Broadcast.com sale. But it was a fair chunk of change for editorial content that literally came out of nothing.

At the time of that sale, I no longer owned the original properties, so I personally didn’t come away with those millions. Darn! And I couldn’t work out a deal with the new owners, so I left and started over. Everything rebooted, from scratch. Enter new site here at Search Engine Land. Entire new conference series, Search Marketing Expo. Our Sphinn news sharing site, as well as our Search Marketing Now webinar series. All through my own Third Door Media company.

All this was done without needing VC money. No various rounds because we actually generate revenue to pay our own way. Last year, in the worst economic downturn in the United States since WW II, we generated a small profit. All from editorial content. News content and information. From visitors we gain through social media marketing, email marketing and search marketing. Google — you know, that vampire.

So yes, I think I understand some of the business elements of running a news operations in the days of the internet. I also know many news SEOs in the newspaper industry who understand the traffic they bring in. I understand how fundamentally screwed up the newspaper industry is when those same SEOs can’t get their technical departments to implement a friggin 301 redirect without debates and justifications. As I joked on Twitter last year:

for want of a 301, a newspaper was lost. or a website. tech folks, give your SEOs the 301 redirects they want.

I’ve also seen newspaper executives — the people who are supposed to save the industry — say things that make me desperately afraid they really don’t understand how search engines work. And if they don’t understand these things, it’s hard for them to make the right business decisions. That’s one reason I invested a huge amount of time on a three part series with Google last year about how it deals with news content. It was important information to get out. It was good information to get out. And I can only hope some of the actual business people paid attention to it.

As for Mark’s hypocrisy telling media owners that Google’s a vampire when he has investments in similar vampires, he wrote:

Danny Sullivan thought he had caught in some hypocritical act because I am an investor in Mahalo, a human powered search engine that leverages SEO techniques to increase traffic and revenue. First of all, I invested in Mahalo in 2006 . Not yesterday as Danny would seem to imply

I didn’t imply that it was yesterday. And I don’t care when he invested in it. He’s still an investor. He seems to disagree with how news aggregators and search engines list media content. Rather than speaking generally to what he thinks media owners should do about the vampire problem, he could use his influence to prevent it within his own investments. And if he’s not listened to, get out of them or speak publicly against them.

Further, he writes:

In the case of Mahalo, unlike newspapers, they are making good money from Google traffic. No reason to stop doing that.  On the flipside however, its fair to point out that Mahalo does use some newspapers content to support their content.  If a newspaper would ask me if they should block Mahalo, the fair answer would be that there is no reason not to.

Unfortunately, those newspapers can’t block Mahalo if they wanted to. Some of those links are added by human editors. Is Mark suggesting that newspapers should constantly be monitoring Mahalo to see when new links are added, then asking to have them removed? Or has Mahalo gained some type of universal “never link to this site” to advise editors, that I’m not aware of.

As for the automatic links, they come from tapping into other search engines such as Google, YouTube and likely some other places that I can’t quite figure out yet. To block Mahalo, you have to block those places. Mahalo, unlike a real search engine, has no spider that can be blocked automatically using a robots.txt file.

1. Lets Google know that you’re interested in that subject.
2. Alerts you to significant updates by putting the headline in bold, so you can get more information.
3. Allows you to follow your 20 most recent starred stories in the “Starred” section of Google News.

Here is a picture of the new stars on Google News:

The changes do make things a bit easier for News publishers though in a couple of ways:

• You can now reference your News Sitemap in your robots.txt file or ping Google with its location, rather than submitting via Google Webmaster Tools (I would still recommend submitting via Webmaster Tools the first time for the benefit of the parsing error information)
• You can now combine articles of multiple types into one News Sitemap. Previously, you had to separate them by genre, access level, and language.

Since you’ve only got a few months left modify the scripts that create your News Sitemaps, I thought now would be a good time to recap just what the changes entail and highlight some of the other changes for Google News publishers with the recent addition of the Googlebot-News user agent.

Remember that you can only submit a News Sitemap to Google if you’re already included in the Google News program. Once you are included in the program, News Sitemaps are a vital way to ensure Google crawls and indexes all of your news articles in a timely manner. If you’re already submitted a News Sitemap, once you make the protocol changes, you’ll need to resubmit it. Make sure everything’s in place before April! Google has an FAQ about the transition that might be helpful.

News Sitemap Protocol Changes

The changes to the protocol are straightforward. Previously, when you submitted a News Sitemap to Google, you selected a publication label from a menu. Now, you simply add this label information directly to the Sitemap (the publication label no longer exists in the Google Webmaster Tools UI). This change paves the way for the two other changes mentioned above.

The new tags (which are child tags of the URL, so therefore should be specified for each news article) are:

<publication>
Includes child tags for the name of the publication and the language of the article. The name tag must match the way the publication name appears in articles, and the language tag should use the ISO 639 language code.

<genres>
Comma-separated list of one or more of the following accepted values (if applicable): PressRelease, Satire, Blog, OpEd, Opinion, or UserGenerated. Note that several of these are new. Google told me:

The only change here  is to simplify the process for our publishers and improve the accuracy of our labeling. The <genre> tag is meant to differentiate between different content types, many of which we use to label articles. Those include press releases, satire, and subscription or registration content. In the past, separate sitemaps had to be submitted for all press releases, satire articles and blog articles from a particular site. This tag allows publishers to submit only one sitemap, and to use those labels on a per-article basis. We have also added additional tags, including op-ed, opinion, user-generated and blog.

Google uses these labels to identify content to readers. For instance, Google News started labeling “blogs” for news readers back in September 2009.

<access>
Subscription or Registration (if applicable).

<title>
Optional tag that enables you to specify the title of the article. You might want to use his tag if Google has had trouble extracting the correct title from your articles in the past.

You can find a full explanation of the current version of the protocol in Google’s help center. An entry in the new Sitemap format might look a little something like this:

<url>
    <loc>http://www.example.com/tv-news/new-buffy-series.php</loc>
    <n:news>
      <n:publication>
        <n:name>Vanessa's World Of Buffy News</n:name>
        <n:language>en</n:language>
      </n:publication>
      <n:access>subscription</n:access>
      <n:genres>pressrelease, blog</n:genres>
      <n:publication_date>2010-01-31</n:publication_date>
      <n:title>Whedon Confirms New Buffy The Vampire Slayer Series</n:title>
      <n:keywords>untruth, wishful thinking, crazy talk</n:keywords>
    </n:news>
  </url>

Of the new tags only <publication> tag is required. Google says the <genres> and <access> tags are required only when they apply, which seems to me to be the definition of optional. I asked Google if they enforce use of them, but it sounds like policies haven’t changed to pay closer attention than before to how things are labeled.

Google News user agent (Googlebot-News), cloaking, first-click free, and subscription content

In December 2009, Google launched a separate user agent for Google News. Previously, Google used the same user agent to crawl for both the news index and web index. As noted in Google’s blog post this change enables news publishers to opt out of being in Google News, but still be found in web search. (Previously, a publisher had to opt either in or out to both.)

Because of the new user agent, and the addition of the <access> tag, I wondered if Google was making any changes to its first-click free program. It always seemed a bit odd to me that while Google was vehemently anti-cloaking, their methods for enabling your subscription-based content into Google News (with the appropriate label) meant that it also had to pontential to be indexed for web search (because the user agent was the same). They told me they had nothing to report at the time, but they have since changed the help center content that describes how to make first-click free and subscription-based content available for indexing in Google News.

The help content now says (about ensuring subscription content can be indexed by Google News):

The easiest way to do this is to configure your webservers to not serve the registration when our crawler visits your pages (when the User-Agent is “Googlebot-News”). It is equally important that your robots.txt file allows access by Googlebot-News.

It previously said:

“The easiest way to do this is to configure your webservers to not serve the registration when our crawlers visit your pages (when the User-Agent is “Googlebot”).”

Google also recently modified its First-Click Free program to enable publishers to limit free access.

Google News recrawls

Just last week, Google announced that they’ll now recrawl articles for Google News. This is great for publishers, as before this, once your article was published and Google had crawed it for the News index, that was it. No changes would ever be reflected in Google News. This could make Google News seems a bit more like print than part of the dynamically changing web. Fortunately, this has changed and Google will now recheck articles it’s already crawled for any changes. Just as it’s always done for its web index.

More resources

For more information on gaining visibility in Google News, see:

• Googler Maile Ohye’s Video Tips For News Publishers
• Google News Blog: Tips For Helping Google News Crawl Your Site
• Google News Blog: Myths and Truths
• Under the Hood: Google News and Ranking Stories
• Danny Sullivan interviews Google News’s Josh Cohen
• Google News Help Center
• Search Engine Land Google News Library

Here’s the statement I received from Google about the AP deal:

We have a licensing agreement with the Associated Press that permits us to host its content on Google properties such as Google News. Right now we are not adding new hosted content from the AP. The licensing agreement is the subject of ongoing discussion so we won’t be commenting further at this time.

Google and the AP signed an original deal back in 2006 that:

• Stopped the AP with proceeding with a lawsuit against Google that would have claimed that listing AP stories hosted by AP members violated copyright laws
• Allowed Google to host stories from the AP on Google’s own web site
• Allowed Google to use AP content for purposes that Google said went “beyond” fair use

Google News Now Hosting Wire Stories & Promises Better Variety In Results covers more about what happened back then when the agreement was struck and the issues involved. Since that time, especially over last year, the AP has been very vocal in saying it wanted a much better deal from Google along with suggestions that despite its existing agreement, Google was somehow still violating copyright.

Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers from November has more background on those allegations and rumblings. Since then, industry watchers have been wondering what would happen as the countdown to the deal’s expiration approached.

Neither Google nor the AP has said exactly when the deal was set to expire. All we’ve known is that it was sometime in late January. February’s since arrived, and since there was no announcement of a deal, it was clear something was up. So I checked with Google and got the statement above.

Where’s AP In Google News? Apparently In Limbo, As Contract Running Out covers how Google has not been carrying AP stories on its own web site since late December. This was apparently done so that if a new deal wasn’t reached, readers wouldn’t be confused about a story they found one day suddenly disappearing the next.

The statement says that Google still has the right to host AP stories on its site but also makes clear that it will not be doing so. Why not? Almost certainly because this is a short term deal designed to buy both sides more time — quite possibly, a short term extension of the preexisting deal. If it’s short term, then Google still might not want to put up stories that could disappear when the short term extension expires.

Not hosting AP’s content also perhaps gave Google a negotiating tactic. It’s now been over a month since Google’s not hosted AP stories, and there’s been no indication that Google News users have complained en masse about them being gone nor that Google News is somehow less relevant without them. In short, Google’s in a strong position of demonstrating to the AP that it doesn’t need its content.

Of course, AP content still resides within Google News another way, available through AP members that carry stories. For example, a search for ap yahoo on Google News brings up this AP story about the AP-Yahoo deal as carried by ABC News:

If a deal isn’t struck, the AP might go back to a lawsuit threat over these types of listings. And Google might decide to stand its ground and declare such usage to be fair use, as it has long maintained. We’ll keep watching.

As for the Yahoo deal, according to the aforementioned AP article, no financial terms were disclosed, nor was the duration of the contact. According to an anonymous source cited, Yahoo does not get any type of advantage over other outlets, something that the AP had suggested might happen in hopes of getting a better deal from one of the major search portals.

For further coverage that’s sure to develop today, see Techmeme.

Postscript: I was asked in the comments below and also by someone else via email why this is a new deal since Google gave a similar statement a month ago. To reiterate what I said above, I think this is likely an extension of the original agreement. That’s why the statements are probably similar — because if it IS an extension of the original agreement, then the basic rights allowed under that haven’t changed. But the original agreement was to have expired at the end of January. Google has confirmed that with me. So something was reached to allow Google to continue forward with AP content — either a completely new deal or a short term extension of the old one. Since the AP isn’t talking about a new deal as it is with Yahoo, I think the safe assumption is a short term extension.

Postscript 2: Reading further about the Yahoo deal, I came across this in the AP story about it:

Besides pumping Internet companies for more money, the AP also wants more cooperation in its effort to ensure its material isn’t appearing on unauthorized sites. As part of its crackdown, the AP is testing a system that tracks where its stories are being read. Yahoo pledged to enforce “the strictest standards” to protect the AP’s content.

I’ve tried for some time to get the AP to explain more about the details of this tracking system it plans but had no luck. I’m optimistic that might change in the future. In the meantime, that system seems to be based on getting AP partners to install tracking code on their articles, so the AP can monitor authorized usage.

As for unauthorized usage, well, unauthorized users aren’t likely to install AP tracking codes. Instead, the AP is likely to do keyword-based searches to find web pages that it considers to be using AP content beyond that which is allowed by fair use. And since the AP has suggested that even quoting an AP headline might be beyond fair use, almost anything citing an AP story might be deemed unauthorized.

That leads back to Yahoo’s statement. The AP has lobbied for the idea that search engines (Google in particular) should give AP content a ranking boost over other content (see Sorry, Tom Curley: Don’t Expect A Google Ranking Boost For The AP). So is the AP getting that from Yahoo in this deal, a ranking boost or some pledge that Yahoo will wipe out anything the AP deems unauthorized content?

I put that in a question to Yahoo, asking the company:

Can you shed any more light on what this means. I mean, is Yahoo not trying to protect AP’s content already? What exactly new is going to happen?

In response, I was sent this statement:

Yahoo! is focused on providing consumers with the most comprehensive and personally relevant content experiences available on the Web.  AP is an important part of that effort and remains one of Yahoo!’s most valued content partners.  We look forward to continuing our long-standing partnership with AP for many years to come.Deal Message Points:

• AP will continue to provide Yahoo! with news content globally and across desktop and mobile platforms.
• Yahoo! will continue to enforce the strictest standards for AP’s licensed content.
• Yahoo! and AP will continue to explore innovative new ways to leverage the power of social and local media, AP’s world-class reporting and Yahoo!’s engaged audience to deliver the best content experiences on the Web.

I bolded the key part. “Continue to enforce” suggests that there is nothing new here. Meanwhile, “strictest standards” means nothing at all. Whose standards? And standards for use on Yahoo or off?

In reality, I think the AP isn’t looking for a ranking boost but rather wants to ensure that if there’s an AP story that makes it into the top results at any search engine, that the story is hosted by the AP itself or an authorized AP member. That’s reasonable. As I wrote before:

The AP seems to want to ensure that if one of its stories is managing to get into the top results, that the AP itself gets the spot, not a submission of the story over at Digg, not a summary of the story over at the Huffington Post, not a copy of its story on one of its many member publications…

That’s actually more reasonable. In fact, SEOs have long been lobbying Google for ways to ensure that original source documents show up ahead of pages that simply reference those documents with little value-add (IE: news flash AP, this isn’t just your problem, and people have been actively working long before you to help solve it). One solution that came this year was the canonical tag, which is about to expand with cross-domain support.

Another solution remains with the AP itself. By not having its own news portal, by having stories that can disappear after 30 days, it constantly shoots itself in the foot to gain the links that would let it naturally rank better in Google.

As far as I can read between the few lines of the Yahoo statement, the AP is NOT getting a guarantee of this from Yahoo. Indeed, with Yahoo set to sell off its search technology to Microsoft, it’s likely would become even harder for Yahoo to make such a guarantee.

The report is News Users 2009, produced by Outsell and based on survey conducted in July 2009 to measure news consumption habits in the United States. It involved nearly 3,000 US consumers.

Search Engines: Growing As “First Thing In Morning” News Resource

One question surveyed how people got their morning dose of news. What did they check “first thing” in the morning? Responses from 2009 were compared to those from a similar question asked in 2006.

TV was the top choice. Print newspapers tied with search engines as the second choice (specifically, Google, Yahoo, MSN, AOL News were listed collectively as a single choice). Unlike TV and newspapers, search engines grew in popularity as a morning choice when compared to 2006. In summary:

• TV: 30% in 2009, down 6 points from 36% in 2006
• Print Newspapers: 19% in 2009, down 4 points from 23% in 2006
• Search Engines: 19% in 2009, up 9 points from 10% in 2006

Radio, the fourth most popular morning news choice, also declined.

Online Newspapers Also Gain Over Print Newspapers

Along with search engines, another news source saw a gain. People were asked about online newspaper reading separately from print newspaper reading. While print declined, online newspaper usage rose from 3% in 2006 to 6% in 2009. That’s a 3 point gain, almost exactly what print newspapers lost.

So are search engine responsible for the decline that print newspapers saw or are online newspapers the reason? There’s no way to know for certain, but I’d expect it’s a mixture of both.

Power Users Like Search Engines & Online Newspapers

The survey also looked at how “power news users” behaved compared to “regular news users.” For morning news reading, power users — who are seen as trend setters — were slightly more likely to use search engines than regular users. This behavior, warns Outsell, is “a darkening cloud on the horizon, portending more shrinkage of newspaper usage.”

However, the report also found that power users were slightly more likely to read online newspapers than regular users. So if they really are trend setters, that stat could also be used to show that the clouds may be parting for the newspaper industry.

Search Engines Win For Breaking News

The report also looked at what sources people turn to for “news right now” — breaking news. In this, TV has been unseated from its number one spot, replaced by search engines:

• Search Engines: 31% in 2009, up 11 points from 20% in 2006
• TV: 30% in 2009, down 13 points from 43% in 2006

From those stats, News Corp. CEO Rupert Murdoch needs to be less worried that Google is somehow stealing users from his Wall Street Journal and more concerned that his Fox News television channel is being threatened!

Radio and print newspaper usage also saw declines compared to 2006. However, online newspapers nearly tripled their share, rising to 8% in 2009. They were the fourth most popular breaking news source, behind third place “Other Online Sites.”

Search Engines Lose For Local News

The survey also looked at how people sought national versus local news. Search engines ranked as the second leading source for national news. But for local news, search engines barely registered. With a 4% share, they were way behind TV — the leading source — at 31%. Print newspapers came second at 30%, followed by online newspapers at 17%.

Do They Really Just Read Headlines?

Probably the most reported stat from the survey has been that 44% of people reported scanning headlines at Google News but then not actually leaving Google to read the full story at newspaper web sites.

That stat came from a press release about the report issued earlier this week. I think that figure needs to be taken with a big grain of salt. In particular, respondents could only choose one answer among four choices about how they used Google to locate news. The choices, prefaced by the percentage that selected each one:

• 44%: Scan headlines on Google without accessing newspaper sites
• 30%: Do not use Google to find news stories
• 14%: Use Google to find local newspaper stories instead of news site search box [IE, Used Google to find local stories by searching at Google]
• 12%: Use Google to find local newspaper stories then use news site search box [IE, Used Google to find local stories, then searched for more at news site]

Personally, sometimes I scan headlines at Google and don’t click on a story, either because I’ve learned enough from the headline to know all I want to know about that story or don’t care to learn more. Other times, I do click on headline links to read a full story. I exhibit two entirely different behaviors. But if I took this survey, I’d have to choose only one. That could turn me into a “scanner” even though I don’t actually scan all the time.

I doubt I’m unique. I suspect many people in the survey who said they were “scanners” chose that because often they scanned, but not always. Indeed, Outsell told me people were asked to choose which answer best described their behavior. As a result, the scanning figure is likely inflated to some degree.

No question, there are headline scanners, perhaps a hefty percentage of them. That type of usage has caused some news publishers to claim that Google in particular — and news search engines and news aggregators in general — are somehow robbing them of an audience they’d otherwise get. Now there’s an actual figure they can cite.

Unfortunately, I don’t think the figure proves or disproves anything. To really determine if Google is pulling from newspaper sites by showing headlines, you’d need to ask about some additional news consumption behaviors.

For example, are people looking for a broad overview of news from multiple sources? That’s something most newspapers don’t provide, because by their nature, they summarize only what they themselves are reporting. If so, it’s hard for Google or any aggregator to steal away an audience that the newspapers aren’t serving.

Similarly, do people preferring the layout and story clustering at Google News versus the display at a typical newspaper web site? If so, then again, it’s like saying that Google is stealing an audience that wants TV shows in color when the “traditional” options only broadcast in black and white.

30% Don’t Use Google To Find News

Meanwhile, 30% of people say they don’t use Google at all for news. That’s a huge number, considering several prominent news publishers have all but blamed Google for the destruction of the newspaper industry. Google’s destroying an industry when 30% of news seekers don’t even turn to it? The number is even higher when you look at “regular users.” Of that group, 37% say they’ve never used Google.

The questions themselves also further muddy the waters. Two choices ask about finding “local” news stories even though the other two choices don’t specifically mention local coverage. Also, Google News itself is also not specifically named. Instead, just Google is. Are these people answering just how they use Google News, which shows headlines by default? Or are some of them talking about how they might search on Google in general for local news?

Ideally, I’d say this question should have either allowed more than one response or had more nuanced choices, such as:

• I never use Google to find news
• I scan headlines at Google but never click on those headlines to read the actual articles
• I scan headlines at Google and sometimes click on those headlines to read the actual articles
• I scan headlines at Google and often click on those headlines to read the actual articles
• I scan headlines at Google and always click on a headline during any visit

Overall, I’d use caution in interpreting this particular question to try and prove or disprove the equity of Google’s impact on newspapers. I think the other questions I’ve covered from the report about search engines and news consumption are more specific and less open to misinterpretation.

As for Outsell’s take on my issues with the Google usage question, analyst Ken Doctor emailed me this:

I checked with Outsell’s survey staff. On this question, respondents were limited to one answer, asked which answer best described their behavior. Such directions are often used in surveying to get to most dominant behavior. Clearly, behavior is nuanced, time-dependent and multiple. My sense is that it is one noteworthy data point, which provides some clues to current behavior and should lead to deeper research.

Thanks also to Outsell and Doctor for allowing me to share some of the additional figures from the report. There’s much more in it, full of fascinating stuff about news usage. It’s available for purchase from Outsell here.

Now, according to PaidContent, a proposed amendment to a pending UK law (”Digital Economy Bill“) would permit search engines to index any or all of the content on a “publicly accessible website” through a “presumed . . . standing and non-exclusive license.” Accordingly, indexing of third party content, however extensive, would not be liable for copyright infringement. This amendment (whose adoption is uncertain) is from self-described Libertarian-Conservative Lord Lucas. But it’s a fairly radical provision.

It’s unclear how using robots.txt would play here in terms of burdening the rights conferred by the amendment, which are much broader than “fair use” in US copyright law.

If the law and the Lucas amendment were to be adopted, imagine a scenario where a raft of new “search engines” appeared in the UK and simply cloned large portions of “publicly accessible” content sites or copied articles wholesale. This might create havoc for publishers, legitimate search engines and end users. It’s unlikely to come to that but such scenarios are clearly implicated by the wide-ranging immunity of the amendment.

The result, which is shown below, was not complete nudity – so we are not sure what type of guidelines Google has against such images in their news index. Google News marked this article in the “most viewed” section of Fast Flip – go figure!

As a matter of note, this is by far the most clicked on image on the Search Engine Roundtable.

We have a licensing agreement with the Associated Press that permits us to host its content on Google properties such as Google News. Some of that content is still available today. At the moment we’re not adding new hosted content from the AP.

So why not? The statement doesn’t explain. But it’s reasonable to assume it’s related to the ongoing talks between Google and the Associated Press.

Google has an agreement to host AP articles on its own web site, plus to make use of AP material in other ways. That expires near the end of this month. Since the agreement only allows stories to be hosted for 30 days, it might be that Google’s covering the legal bases in case a new agreement isn’t reached. You don’t want a story going up on, say January 23, only to have to pull it down the next day.

Google News Now Hosting Wire Stories & Promises Better Variety In Results from us in 2007 has background about how Google began hosting stories from several wire services, including the AP, on its own site.

The AP, in particular, wanted its stories hosted. Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers from us last November explains more about this, how Google makes use of AP content under the current agreement and some of the issues that have come up in talks to strike a new deal.

It’s important to note that AP stories may still appear within Google. Many newspapers carry AP content, and those papers continue to be listed. So you can find AP stories hosted on newspaper sites. You just won’t find them hosted within Google itself.

On a somewhat related note, Rupert Murdoch’s News Corporation has threated to block major search engines, including Google, from crawling its news content. Some see the first shot in that threat being fired as UK-based The Times is now blocking the UK-based NewsNow search engine.

As it turns out, this seems largely unrelated to Murdoch’s complaints with Google. Instead, it focuses on NewsNow providing a commercial service that allows companies to monitor the news. The UK’s Newspaper Licensing Agency wants to charge companies that provide this type of service. While Murdoch’s Times hasn’t joined that push, which is under review by UK authorities, it has restricted NewsNow for the same reasons. PaidContent provides an excellent rundown on the situation in these articles:

• Murdoch Paper Blocks UK Aggregator Before Paywall Goes Up
• UK Newspapers Suspend ‘Link Tax’ Bills To End Users

Technically, NewsNow doesn’t have to obey the restrictions blocking it in the robots.txt file at The Times. It’s not a legally-binding protocol. But respected crawlers do obey it, which is one reason why there have been so few lawsuits over crawling.

That robots.txt file is farcical in one respect. At the top, it says this:

#Robots.txt File
#Version: 0.8
#Last updated: 04/01/2010
#Site contents Copyright Times Newspapers Ltd
#Please note our terms and conditions
 http://www.timesonline.co.uk/section/0,,497,00.html
#Spidering is not allowed by our terms and conditions
#Authorised spidering is subject to permission
#For authorisation please contact us -
 see http://www.nisyndication.com/about_us.html

A robots.txt file is designed to be read automatically by a machine. Anything line that begins with a # symbol is a comment that machines ignore (the two indented lines above without a # symbol are actually part of the line above, just with a line break in this story, so you can read them properly). There’s no way for Google or any search engine to automatically know this file that “Authorised spidering is subject to permission.”

The way they actually know this is if the robots.txt file uses an accepted command to block them — which it doesn’t, except for NewsNow and some smaller crawlers.

I highly doubt Google, Yahoo or Bing have actually asked The Times for permission to crawl them. But I have an email out to The Times and News Corporation to find out. Of course, if any human had tried to follow the link to seek authorization, they wouldn’t have gotten an error. That link doesn’t work.