Hey news publishers. Stop acting as if your content only appears printed on dead trees and tap into the dynamics that the web offers. That’s a blunt summary of advice from Josh Cohen of Google News, from a wide-ranging interview with him on Google’s experiments with new ways of delivering news.

Cohen, business product manager of Google News, says Google has no ultimate solution for the future of news online. It does have a vision of a super personalized news product that tracks someone’s “read state” and keeps them constantly informed with updates. But to turn that vision into reality, it’s conducting a variety of experiments. Some will succeed; some won’t.

The experiments aren’t meant to compete with publishers. Cohen stressed that Google’s not a content play and has no “Hulu for journalism” pretensions. Google’s a technology company, he says, one with tech that it hopes news publishers can tap into.

For publishers to be successful in a personalized news product, they may have to consider the “Living URL” model of stories, Cohen said. Think Wikipedia, written by journalists. And think about how newspapers might learn from a classic Christmas film, Miracle On 34th Street, where sending your customers away might actually make them more loyal.

Below, you’ll find Cohen’s comments on these and other issues. It’s the second part of a three part series, from an interview I conducted last month. Also be sure to read the first installment, Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers.

What’s Up With Hyper Personalized News?

Both Google CEO Eric Schmidt and Google vice president of search product and user experience Marissa Mayer have talked about the concept of a hyper-personalized news system that remembers what you know, finds what you want to learn about and would be as easy to use as flipping through a newspaper or magazine. It’s been raised as an idea by Google as far back as 2007. Is this real? Is it coming? Is Google working with publishers on it? Cohen wouldn’t say much:

The short really unhelpful answer is, sort of. I think if you look at a few of the things we’ve launched in the last month or so, and launched being a broad term for publicly available, you get a sense of how we’re approaching some of this.

Journalism Should Be Written For More Than Print

Continuing, Cohen spoke more broadly of the idea that journalism still largely acts as if it is meant for publication in a hard-copy newspaper instead of also appearing in the instant access, hyperlinked world of the web:

News online by and large hasn’t adapted to the medium and still is largely brochureware, where people are taking not only just the physical article that was in the paper and just putting it online but also just the way you tell stories.

It [storytelling] just hasn’t really adapted and taken full advantage of the medium. Beyond just the challenges that go with it, it’s a completely different way of telling a narrative. And so there hasn’t been that real transformation. As a result, you get a lot lower engagement online.

Fast Flip: To Combine Best Of Online & Offline

Cohen said that Google Fast Flip, launched in September, is one example of how Google is experimenting to combine the best of both worlds, offline ease of reading and browsing with online’s “smarts” that allow for personalization:

The premise behind Fast Flip is really a step in that direction of trying to figure out how do I create a good online reading experience that is engaging, that captures some of that browse experience?

Fast Flip is focused right now on a specific set of content, more of the longer form content that’s more suited to a magazine, which is why you can sort of get that idea of a custom magazine that you can sit back and browse, flip from one page to the next, and without having to take 5 to 10 seconds for each of those pages to load.

You literally have that offline browsing experience with all the advantages of being online, customization, aggregation, and all those types of features.

Fast Flip: A Test, Not Google’s Ultimate Solution

However, those who interpret Fast Flip as “Google’s plan to save newspapers,” Cohen said, are making a mistake. It’s not perfect. It’s a test, and a test that might not work for everyone or at all.

What it is, is Google’s attempt to try and experiment in one of those categories. I think Fast Flip has a lot of advantages for, and this is my personal view, a certain set of content. How does that work for all forms of news, for all forms of content, I think remains to be seen.

We want to put that [Fast Flip] out there in Labs [Google Labs, where Google releases experimental products]. We’re excited about it. I think the response has been really positive both from users and from publishers. But we want to test it.

I think we all see it as one step in that direction of what does a news experience look like? Could that be a Google-hosted experience? Can that take place on a publisher side of it? Sure, I mean yes, the answer is ‘yes’ to both those. But that’s, more than anything, it’s an idea of ‘Look, we’re experimenting with this.’

Continuing, Cohen explained that Google sees certain issues with how news is delivered online, has thoughts on how things might be improved, and that Fast Flip is meant to test some of those theories:

That’s kind of how we approach things. We go out there and test it. And we have certain assumptions that are behind Fast Flip, which the data will prove us right or wrong, and then we’ll go from there and either continue to expand it and see what’s working and what’s not and iterate on it, potentially blow it out, deeper integration into [Google] news, separate products. Who knows? We try and be as open as possible to letting the product and the users’ reaction to it drive that direction more anything else.

In fact, Google has just begun a small test of Fast Flip being integrated directly into Google News.

Google To Publishers: Can Our Technology Help You?

How about that laundry list of ideas that Google provided to the Newspaper Association of America, which approached companies it selected for ideas on how to monetize news content? Cohen said this wasn’t a specific plan but rather a list of off-the-shelf technologies and systems that Google already had which might be adapted:

The way that we intended it was a response to say, ‘Look, this is our thinking on the space.’

Continuing, Cohen explained:

“We really tried to map out how we saw that ecosystem working [such as news publishers trying to process micropayments for content], how some of the existing Google technologies could potentially plug in to that, and we could power a site in the same way that Google Maps can power your mapping solution. We don’t create content but we create technology, so how can you use that?”

After the document leaked out, Cohen said it kicked off a number of discussions with publishers wanting to know more. Focusing on the micropayment aspect, he commented that subscriptions can and do work, but an easy way to charge is vital

For a certain set of content, and for a certain type of publisher, subscriptions, not only can they work, they do work, for a subset today. Is it a panacea for all forms of content, simply to put in a paywall? Personally, I don’t believe so. But again, if there’s a set of content that it works for, you’ve got to have a technology solution that works.

If you’ve gotten over the hurdle of someone saying ‘Yeah, I’d be willing to pay for this, I’m going to take out my wallet,’ don’t have them go through 50 leaps to get there. Then it’s really doomed to fail.

Google News & Personalization

Cohen then circled back to my original question, about how all these news experiments relate back to a more personalized news experience that Google espouses.

“We always talk about delivering the right results,” he said. “So yes, Google’s working on this. What forms that might come out in, could it just be a better personalization of Google News or certain types of feeds you can subscribe to or custom sections that are there today? I think it’s kind of any and all of the above.

By the way, since I conducted the interview, Google rolled out Custom News Sections, a way to personalize Google News so that you can browse stories that match particular keywords you’re interested in. A form of this has existed since 2007, but the update rolled out a directory, so that Google News readers could share custom sections with each other, along with the ability to have more complex keyword matching.

Challenge Of Explicit & Implicit Personalization

Of course, Google also has personalization that learns through watching behavior. But both explicit and implicit personalization features have much room for improvement, Cohen said.

I don’t think it’s where we want it to be. Personalization is hard to do.

The easiest of those are explicit personalizations. Even that, I think, is hard to get right, because a user will come and make certain decisions or choices about the type of content that they want to see, and yet if they miss something because of the personalizations [that they did], it’s still your fault.

You know, ‘These are the sources I want to see, these are the topics I want to see,’ and then they miss something, and, ‘Why didn’t you tell me about that?’. And again, that’s with a lot of direct signals and instructions from the user.

Then you continue down that spectrum on the implicit side of it, of understanding my reading pattern, not only what stories and sources and topics that I’m interested in but also my experience reading those specific stories. And so I think Eric’s talked a little bit about this, the ‘read state.’

Read State: What Online Can Detect That Offline Cannot

Read state, Cohen explained as he continued, is a key challenge that Google feels online news faces, the need to figure out where someone has left off in following a particular news story.

Think about it this way. The traditional newspaper reader would get their morning paper, read some stories and be done. The newspaper had no idea what they read. So when writing updates to those stories, the newspaper was forced to assume you knew nothing. It had to get the most important breaking aspects up at the top of a story, writing in “inverted pyramid” style so that if a reader drops off, the less important facts are safely buried further down in the story.

Online papers could be smarter. They could understand what you’ve read, where you left off and keep you informed with only the new material you need, because they’d understand your read state. But online news isn’t written this way. It continues to be produced as if people are reading offline. As Cohen explained:

Every single day I have to put something out in the paper. So there’s an on-going story. Every single day I file another article. A deadline comes in, 6, 7 o’clock or whatever, I file it, and it goes out there because I have to put something out there in the paper.

As a result, often times you have to have a certain set of facts, even if they’re just one little update to that story. It generates a larger story either to fill space or because I can’t just put a quick headline update to it and link back to my other sources to it.

Part of the reason that you see that is, one, there hasn’t been too much innovation in the space. But also because people don’t take into account your read state. So if I know that you’ve come there, I can give you the full story, or I can give you a quick update, a bulletpoint summary. That’s another level of personalization that I think is not there.

Living URLs & Bringing “The Miracle On 34th Street” To news

In theory, the idea sounds great — this “Living URL” idea that Google’s Marissa Mayer has especially suggested, the story that lives in a single place, constantly being updated. But aside from current technical issues Google News has, where it can’t even handle stories like that (see Of Living URLs, Newspaper Rankings & California Fires), there are current business issues that prevent it.

For example, what happens when one paper reports on a story, then a different publication reports a unique and specific fact. You do you merge what the two competitors say together, especially when they often don’t want to acknowledge each other?

That gets to another issue, people not linking out. Publishers trying to be all things to all people, instead of a focus area, of whether it’s a regional area where it’s the LA Times and I covers LA or it’s a topic where I’m the Washington Post and I can cover politics or I’m the Wall Street Journal, and I can cover business…

My value is my editorial filtering. I recognize that if I send you off [my site], and I just put a link to an update, if I’m the New York Times, and the Wall Street Journal has a good update, I’m going to link off to it, so that I know that this is a good source that tells me what I should be reading, even if it’s not on their own site.

That’s what’s going to set me coming back. Not because I’m trapped into their web site, and I have to know that all the information is coming from there. There’s a comfort level, there needs to be a comfort level, to send people out.

That’s certainly a Google bias towards this. Where our focus is largely on getting people off of our site, because we recognize if we provide value in serving them the most relevant search results, whatever it is, news results, that this is going to be a jumping off point for them.

Call it the Miracle On 34th Street approach to news. For those unfamiliar with the classic film, the Macy’s department store Santa Claus (who is actually the real Santa) sends shoppers who can’t find what they want to Gimbles, Macy’s archrival. While at first Macy’s management is horrified, they’re won over as a customer declares her loyalty to Macy’s for putting the customer first.

In my own experience, the mainstream media traditionally has not linked out. It’s more common where they have blogs, such as Bits from the New York Times or Digits from the Wall Street Journal. And among news blog, it’s very common to crosslink (my Blogs & Mainstream Media: We Can & Do Get Along post gets into this more).

Should Newspapers Become Like Wikipedia?

The Living URL / Read State concept also sounds similar to something that already exists: Wikipedia. Should papers all simply become Wikipedia-like, where stories about a particular topic reside on a single page that’s constantly updated? Cohen’s not certain himself and figures there will be lots of experimenting.

I don’t think we have that answer. That’s something else that I think we’ll certainly experiment with on our side and experiment with publishers directly. I think the concept makes a lot of sense. How do you put that theory into practice?

I know you played around with certain parts of it, and you felt certain parts didn’t work felt or not. That was like one take at it. There will probably be 50 different takes at it from a number of different sources….

I think everyone recognizes the value in an online world, of having that persistent URL and a single source to get all the updates on a given story. That doesn’t mean there aren’t changes that need to take place on the editorial workflow, on the product design and also on the search side of it about how you pick those things up, too, which I think you pointed out.

Is A Living URL System In Testing?

But does Google have something like this in testing, that hasn’t been put out there? What about all the rumors that Google has something in the works with the New York Times or the Washington Post? Cohen wouldn’t say but rather pulled back again to stress there’s no perfect grand plan system Google has up-its-sleeves:

I will say this. We are directly working with a number of different publishers. We talk about the monetization, but product and content creation as well, whether it’s a set of tools or a structure or do so.

There are definitely discussions we’ve been having. There was that story that was reported about these collaborations with the New York Times and the Washington Post. It took bits and pieces of a lot of these different experiments and turned into a brand new product that would be the “future of news.”

While it is accurate that we’re working with the Times and the Post – among others – about improving the online news experience and how Google might contribute, as with many things, it often gets turned into something much grander.

Google Unlikely To Build A Hulu For Journalism?

How about the thought of a “Hulu” for newspapers and other journalism outlets, where Google might compile all the stories together for publishers, with single monetization or a subscription basis?

Cohen didn’t see Google as putting out a specific collective solution but rather staying focused on publishing tools that publishers could use individually.

Broadly speaking, the area of creating platforms for content is something we do today, such as with Blogger or YouTube or Knol. These are all examples of Google trying to make it easy to put content online. That’s consistent with what we want to do,” he said.

But Fast Flip seems like the start of a Hulu. Or is it something that Google may license as a tool for anyone to use? Indeed, that might be its future.

“Fast Flip is a longer term vision. We’re not saying it must be hosted at Google,” Cohen said. It might evolve into something publishers can host on their own sites or customize, he said.

Spotlight Added; Newsmaker Comments Go – Learning From Everything

How about the new Spotlight section of Google News. What’s the purpose behind that, and what’s selected for it?

It’s been really successful with us so far. There’s interest from publishers that it has potential to give another platform for content that often doesn’t do that well with Google News, which changes hourly. It’s for long shelf-life content, the enterprise stuff, the investigative pieces.

And thoughts on things that have gone away, such as dropping the attempt to get quotes from newsmakers on stories. Disappointing?

I don’t mean this to sound too Pollyannaish. Google is trying to encourage itself as a place where you can have failures and think more radically about the approach to a product. There’s no sense that I have get this perfect or it will never launch.

The comments feature is something we killed, but we also learned something from it …. People are encouraged to ask ‘What did I learn from this?’ That’s what makes it interesting to work on the products.

Are there things Cohen particularly likes? Cohen said he was happy with many “under the hood” changes that he thinks help people better trust Google as a news aggregator.

I was curious if more people browse stories at Google or do keyword searches. Cohen wouldn’t give specific figures, but he did say that at a typical news site, he knows that searches tend to be in the single digit percentages. In contrast, browse versus searching at Google “is much more balanced,” something he didn’t think was surprising, given that ultimately Google’s a search engine.

Google News: Trying To Serve A Balanced Diet Of News

Finally, I’ll end this part of the interview with what Cohen said was the overall mission for Google News, to “educate and inform.” In particular, Google News is aiming to expose things you want to see alongside things you should see. Or as Cohen put it, “serving the vegetables as wells as the dessert.”

Next week, the final part of this interview, looking at how Google News determines what to show visitors.

The Google Mobile Blog announced that Google News on the iPhone, Android and other smart phone devices has a new design. The new design looks more like the desktop version of Google News with minor changes to make it fit better on the smartphone screens.

The new design has more stories and images, has personalization features, search and a neat “jump” to section feature. Here are screen captures:

Many publishers have complained repeatedly that Google and Google News (not the leading online news site) reduces their content to “commodity” status or otherwise adversely impacts their brands. Google’s Fast Flip is a response to this in part.

It offers a more “branded” and visual way to consume news and magazine content from major publishers. We wrote about Fast Flip originally when it launched in Google Labs. Now Steve Rubel has found a page that looks like Google is getting ready to integrate Fast Flip into News:

It appears from the screen above that Fast Flip might replace some part of the news page below the fold. I like the concept of Fast Flip quite a bit but think it still has a way to go to be a compelling way to consume news.

We also speculated some time ago that Fast Flip might be the “platform” for the micropayments system that Google has been working on with a number of newspaper publishers. In the meantime, Fast Flip offers Google some valuable “inventory” against which it can serve contextually targeted display ads:

There’s also a mobile version of Fast Flip:

Fast Flip is a product that has the potential to make Google News different than its competitors and much more interesting than it currently is today.

Postscript: Here’s an official comment from Google:

At Google, we run anywhere from 50 to 200 experiments at any given time on our websites all over the world. A few weeks ago we started running a few small ones exploring how we might incorporate results from Google Fast Flip, the article-reading service we launched in Google Labs in September, into Google News. From these tests we hope to learn whether including Fast Flip results in Google News would provide a good experience for users and news publishers. More information about how Google runs experiments can be found here.

Want to do a paywall with no “first click free?” That’s fine with Google, says business product manager Josh Cohen. Want to do micropayments? Google will be “flexible” in considering support of new business models like this. But if you charge, expect less traffic, and also expect that your competitors will be “ecstatic” to pick up your loss, he said. Cohen’s comments on paywall issues were part of a wide-ranging interview I had with him about Google and its news service.

In the interview, Cohen also repeatedly stressed that publishers are free to deal with Google as they like. And if they wanted to exclude Google in favor of a competitor like Microsoft, they have that choice — though Google would prefer to work with everyone. Indeed, Google’s inclusion of many diverse sources is one reason he thinks Google might take hits from certain publishers that don’t criticize Yahoo News, which is a far more popular news service than Google News.

Below, you’ll find Cohen’s comments on these and other issues, along with a summary of how Google handles free, registration and subscription-based content. I spoke with Cohen in early October (a busy month in search has kept me from getting this posted until now). His comments are even more relevant given that talk of news publishers perhaps blocking Google have only ramped up in the past weeks.

Paywalls & Google Are Not Mutually Exclusive

As publishers increasingly consider putting up paywalls (a barrier to reading a story unless you’ve paid for it in some way), I also see commentary from others arguing such moves would be dumb, that they’ll be the final nail in killing some publications. What’s a confused publisher to do? For me, I find it puzzling publishers believe they have to make a choice. They can have their paywall AND Google traffic combined, via Google’s First Click Free program. Are there many publishers who simply aren’t aware of this program? Cohen responded:

Yep …. I’m often surprised, and maybe this means I’m not doing my job particularly well, but there are some basic questions on First Click Free and support for subscription content or the rankings or what we try and do.

Some of that are the challenges that you can’t make everything widely available. But we do try to be fairly transparent with what we’re trying to do within our algorithm on the Google News side of it, because we recognize that it’s important to different publishers. We want to be in a situation where the best content wins, not the best SEOed site. So if we can put that out there as much as possible and in essence give all publishers a level playing field, then the user wins.

If a publisher understands that quality, original content does best and therefore tries to create more, great. But there is still a lot of those discussions that take place where people will say … ‘I have to make this content free or Google won’t index it,’ and that’s not the case.

First Click Free is only one example of the ways that publishers can make subscription content available. They can do previews, they can block it in different ways. I think there are a lot of those questions about the nuts and bolts of how you can work with us, subscriptions just being one of them.”

Paywalls Don’t Require That The First Click Is Free

Of course, one concern with First Click Free is that it can allow a savvy person to effectively read your publication for free, as can happen with the Wall Street Journal. In fact, News Corporation raised this as an issue this week (My Would Someone Please Explain To News Corp How Google Works? article looks at News Corp’s comments in more depth).

That’s the first time I’ve seen a major publisher talk about issues with the First Click Free “backdoor,” but it was something I was expecting to come up, so it was on my list of questions I’d put to Cohen. Can publishers be listed in Google News and still protect their paywall? Cohen said yes:

You can allow us to crawl content and show a preview to the user and label it as a subscription. So that happens today. You can do that in Google News today …. as long as you’re not cloaking [showing Google something different than what a visitor would see, when they finally arrive at the page they've paid for].

You can also show a preview and we can index it, as long as it’s a consistent experience, where if you’re showing us a preview, and we can index that, and that’s what you show the user, that’s fine too.

How Google Handles Free & Paid News Content

In summary, here are the basic ways Google News handles news content that is free, requires free registration or requires payment to view. These are also described in Google’s help pages:

Free Content: Content is free. Google can index an entire story to make it searchable. People can find the story in Google and read the entire thing for free.

First Click Free: Content is behind a paywall. Google is allowed past and can index the entire story to make it searchable. People can find it in Google and read the entire thing for free. From that story, people cannot click to read other stories at the same publication for free (hence, the “first click free” name). However, people can potentially go back to Google, find another article from the same site, click to it from Google and read that.

Subscription: Content is behind a paywall [or requires free registration to read]. Google is allowed past and can index the entire story to make it searchable. People can find it in Google. They can only read the entire thing if they pay [or register].

Preview: Content is behind a paywall [or requires free registration to read]. Google is NOT allowed past to index the entire story to make it searchable. People can find it in Google only based on the preview content. They can only read the entire thing if they pay [or register].

And because I like charts:

Clearly, there’s room for improvement. Flagging content that can be viewed through free registration as the same as content requiring payment doesn’t make much sense. It’s also unclear to me if the “subscription” label means much to someone viewing search results. Perhaps “Pay Per View” speaks better to what they can expect.

The Preview option may be hard to comprehend for some publishers. It seems designed for those who are absolutely paranoid that they don’t want Google to index their content, even though giving Google only a summary story degrades the chance of performing as well in search result. The subscription option increases their findability yet protects their content from the general public just as well.

No Subscription Option For Regular Google Search

Note that the options above also are only for Google News. In Google Web Search, the Free and First Click Free options are allowed. Publishers can manually do Previews on their own. But there’s no option to show subscription-based content in Google Web Search, with the exception of Google Scholar content.

That’s a big deal. Newspapers get tons of traffic from regular Google web search. If they go subscription-only, they’ll lose that traffic. That might be fine for News Corp, which argues that the traffic it receives isn’t that valuable (despite, oddly, also purchasing ads on Google to gain more traffic). But the difficulty for Google in allowing subscription content into regular web search is that if the top results get flooded with it, uses may become dissatisfied and express their frustration on Google.

Open To Listing Arrangements

Some of how things currently work may change, however, especially as the news industry itself is attempting new business models. Said Cohen:

If people are putting more and more behind a paywall, it’s in both parties’ interest to be as flexible as possible around that, provided you can maintain a good user experience.

For us, we obviously want to be able to index that content in a reasonable way for our users. If we can find ways to be flexible in supporting whatever models come up next, whether it’s micropayments or whatever else that may be on the horizon, that’s good for us.

For [publishers], there is still a recognition that discovery is really important. I would argue even more important if you’re putting content behind a paywall, because all of a sudden, depending on your model, again, you’re potentially shrinking your potential base of users. So you want to increase the size of that funnel, you don’t want to restrict it even further.

That’s why we have those discussions with publishers. That’s why we work with the Wall Street Journal, the FT [Financial Times] and others with their subscription content because again, both of us are trying to do the same thing, which is make sure their content can get found.”

Subscription Content Doesn’t Rank As Well

Of course, while publishers are free to go subscription-only within Google News, they risk having lower visibility if they do so. Cohen explained:

The reason that subscription content won’t do particularly well in search results is just the user behavior. I’m not saying all information wants to be free and has to be free, but the user behavior is by and large that people don’t pay for a lot of that content.

If you have subscription content, the user response to it will in effect tell the algorithm this isn’t not a relevant result, I’m not clicking on this. By making it free or by in essence saying it’s paid but Google treats it as free [because of First Click Free], there’s a significant advantage to them, because all their content is indexed, and I think at the end of the day probably helps the results. People are more likely to link to it and all the different ways it can be beneficial.

It’s not that the Google News algorithm treats subscription-content as second-class just because it is flagged subscription. Instead, Cohen clarified, it’s that the algorithm tries to mirror what users like. Since they largely bypass subscription content, less of this is surfaced. I’ll also have more about the Google News algorithm in general in a future part of this interview (see the end of this story for more).

The Many Shades Of No

Cohen also stressed again that the choices remain with publishers. Beyond the Free / First Click Free / Subscription options, they can opt-out entirely from Google News or Google itself. They can also say no in a far more granular way than some like those backing ACAP (a proposed next-generation blocking and access system for search engines) would have you believe:

There are discussions saying ‘You’re stealing my content,’ but publishers have complete control on whether that content goes online in the first place. The publisher’s in complete control about the business model. If they want put up a paywall, again, publishers can put up a paywall. We don’t force you to make it free. In fact, we work with a number of publishers today who charge for content.

The other part of the extreme is even if you’re online, that doesn’t mean that we can come in and force you to index your content with us. And this is the whole robots stuff, where if you don’t want to put it in Google, or even just in Google News, you can block it, you can segment it or if you don’t want to show snippets. If you don’t want to show images, you can do that too. The publisher has complete control about whether that content is displayed. [By "robots stuff," Cohen is referring to the robots exclusion protocol, explained further below in this interview.]

We certainly hope people don’t opt out. We think there’s high-quality content, and we want to be able to index it, but at the end of the day, the publisher has control over that. So this sense that they have no choice but to be in Google, that they’re forced, that we’re breaking into their house and taking that print, digitizing it, putting it online and forcing it to be free …. There are so many steps along the way where they can say ‘Stop’ and we will respect that, 100%.

Build Your Paywall & Others May Go Elsewhere

How about the idea that newspapers all need permission to collude, to discuss openly banding together blocking all their content, because if they don’t all do it at once — or if one person stays out offering material for free — then collectively they all lose. From that comes a suggestion that they really don’t have a choice, that they have to be in Google. Cohen disagrees with this.

They do have a choice.  I think the reason publishers want to be in Google is because of the value we deliver.  There are a number of sources of information competing on the web today, so making sure your content is discoverable is by and large a good thing.

This idea that ‘Oh, so nobody’s going to read any other source other than what’s in a newspaper.’ Think about that for a second …. you have a number of different sources out there that are non-newspapers who are probably just ecstatic at the prospect of a lot of paywalls going up in a lot of different categories.

You know, pick a category. CNN, general interest news, for example, I’ve got to think, and I don’t know, I don’t know anything, I’ve got no insight into CNN’s thought process and maybe I’m wrong, but they probably get a ton of traffic and do a fairly healthy business on the online side of things.

So if all these newspapers go behind a paywall, I would have to think that somebody like them, who’s in a strong position right now, is probably going to take a different position.

Want To Exclude Only Google? That’s Fine

That led me to the other solution that gets floated out there, a sense that there needs to be either an improved “hot news” law or tighter restrictions on fair use, so that people cannot so easily summarize stories (such as when a blogger does highlights of a news story or when a mainstream news source summarizes a story from another mainstream publication).

When the interview happened, the AP has just suggested it might give its stories in advance to certain portals — widely interpreted that maybe Bing could get a head-start with news stories over Google. What did Cohen think of those lobbying for fair use changes and the AP’s push?

Cohen didn’t really answer the fair use portion, though Google’s been pretty clear that they feel what they do falls under fair use. As for exclusivity, he didn’t seem bothered if someone wanted to partner with people other than Google.

There’s a challenge over these existing business models. I think some of it is, these are businesses that had 20, 30 percent margins. And that’s probably not going to be the case going forward. I don’t see that just on the business side, forget both the cyclical and the secular changes to it.

Again the reality of it is that the publishers have complete control over their content on the web. Whether or not you digitalize it, whether or not it’s paid or free, and whether or not Google in particular has access to it. With robots, you can specify by user agent.

If you want Yahoo and Microsoft, and you want to do a deal with Microsoft, and only have Microsoft have access to your content and say Google, don’t index my content, you can do that.

The day after my interview with Cohen, Google CEO Eric Schmidt got asked about the AP news, as well, saying that Google had to be careful not to favor one publication over the other in terms of speed or latency.

Excluding Google News Vs. Excluding Google Web Search

One issue that came up with some Italian publishers recently what the suggestion that if you opt-out of Google News, you can’t be included in Google Web Search, which generally seems to send more traffic to publishers over time that Google News does.

So, the allegation was, if you object to being in Google’s news portal that you view as a competitor, you also have to exclude yourself from also being in web search results that you might not view as so much a competitor (see Debunking The Italian Newspapers’ Antitrust Allegations Against Google for more on this).

Not true. The reality is that you can opt-out of Google News but still be included Google Web Search. However, you can’t do this automatically. You have to request being dropped from Google News (which, by the way, also doesn’t automatically include anyone. Some human reviewer at Google decides to include a source, or sources have to manually request inclusion if they’re not already listed).

Shouldn’t the Robots Exclusion Protocol options (robots.txt files or the meta robots tag) used to signal automatic exclusion from indexing allow you to say no to Google News but yes to other Google search properties, such as Google Web Search? That’s not easy, Cohen said, but it could come:

We recognize with Google News, it’s very important to publishers to give them that option, if they want to opt-out specifically from Google News. We allow them to do that, simply by telling us to remove them. But as you apply to a whole set of different services within Google, it just gets hard to define what is the use that’s intended for a given service. That’s the main issue from our side.

But at the end of the day, we want to give as much control to publishers over their content as possible. So if that’s something that we can do in a way that works for users in general, there’s not a business reason why we wouldn’t want to give that control.

Google’s Inclusion Of Many News Sources May Draw Attacks

I also wondered why Cohen felt publishers seem to attack Google more when Yahoo News still outdistances Google as a leading news site by more than 3 to 1, not to mention that unlike Google News? In part, he seemed to suggest that ironically, it’s because Google’s trying to show a diverse and level playing field of sources. It includes a lot of publications and sends many of traffic to many of them, rather than a select few.

I don’t know that it is Google News versus Yahoo News. Probably, they view it more broadly as Google overall and Yahoo overall and getting back to size questions. But I can’t answer that. I’m not in a position to do so. It’s more of a question from the publisher side of it.

From the news site of it, this is not a value judgment, just Yahoo News and Google News have just different approaches and different business models. We’re both aggregators, but they’re more of a portal. Their focus is on creating a Yahoo-branded experience, editorial voice, business model [unlike Google News, Yahoo News employs journalists to produce original content]. They do send traffic to partners but a large part of it, that experience takes place on Yahoo’s site.

What we’re trying to do within Google News is around having as many different perspectives as possible on a given story and a diversity of sources. Also, again keeping with how we tend to operate as a company, our business model is about directing that traffic back out to the publisher’s site. It’s their business model, it’s their look-and-feel, their editorial voice. Again, I honestly don’t mean that as any sort of a value judgment. It’s just different approaches to the space.

Google News & Content Partnerships

Would it help if Google partnered more with content owners for licensing agreements in the way that Yahoo apparently does? Cohen replied that Google has “thousands” of partnerships. But to me, most these are down to AdSense, agreements to place Google’s ad serving code on partner web sites — where the “partnership” has little more to do with someone filling out a self-serve form. What about partnerships that cover the use of material on Google’s site itself, such as how Yahoo licenses out wire content.

Cohen said that Google has 11 partnerships like this of its own, at the moment. After the interview, I was sent the full list:

• AP
• AFP
• UK Press Association
• Canadian Press
• EFE
• Lusa
• Keystone
• APA
• PAP
• MTI
• ANA

A deal with the AP was made in 2007, followed by agreements with AFP, UK Press Association and Canadian Press in 2008 (see Google News Now Hosting Wire Stories & Promises Better Variety In Results for more). The latter seven listed are all members of the European Pressphoto Agency and signed an agreement with Google in March 2009 (see also here).

Belgian papers that fought a much publicized battle against Google in 2006 were reincluded in 2007. At the time, this was not due to a financial settlement (see Belgian Papers Back In Google; Begin Using Standards For Blocking). As they don’t appear on the list of formal partners I was sent, I assume there still remains no formal agreement for their inclusion. I also don’t see on the list a deal with Sofam & Scam, two Belgian photo and A/V services that joined the case after it started, got an agreement, then dropped out of the suit. So I’m not sure if this is still active — I’m checking on this.

Why Do Deals? Reducing Duplicate Stories One Reason

The content deals have been stuck for various reasons, Cohen said. With both the AP and AFP, those deals were designed in part supposedly to help with the problem of duplication of wire content on Google News. The same wire story might appear in various newspapers at the same time, which can confuse Google into thinking the stories are all different, when they’re the same. Said Cohen:

If you’re trying to show different perspectives, having 50 copies of the same story doesn’t make sense.

Continuing, he also talked about why as part of the deal, Google agreed to host AP articles on Google itself:

They have a different business model where the focus [of the AP's web site] is primarily on the B2B space. They don’t have a [consumer] destination. They don’t want to, or they can’t, because of the challenges that might cause for them with sales.

By sales, Cohen refers to the issue that the AP both gives its own original content to member publications plus take stories in from them to distribute to all AP members. If the AP creates a news portal of its own, it potentially competes with member publications for readers, using content from some of those members.

So with the AP, the existing deal was intended to help solve dual issues, that of Google wanting to reduce duplicate stories and for AP to have Google host its stories in a somewhat “neutral” environment that might be more acceptable to its members, since the AP couldn’t host the news itself.

AP’s Ranking Boost Quest

Interestingly, one of the AP’s top concerns with a new deal with Google appears to be that it wants to rank better in Google’s results. Unfortunately, the AP shows its ineptness in understanding search when it speaks like this. There is no way — no way — that Google’s going to guarantee the AP a ranking boost over other news sites.

Really, what the AP seems to want to ensure that if one of its stories is managing to get into the top results, that the AP itself gets the spot, not a submission of the story over at Digg, not a summary of the story over at the Huffington Post, not a copy of its story on one of its many member publications. As best I can tell — I’ve not had luck getting the AP to talk to me directly, but I’ll be trying again.

That’s actually more reasonable. In fact, SEOs have long been lobbying Google for ways to ensure that original source documents show up ahead of pages that simply reference those documents with little value-add (IE: news flash AP, this isn’t just your problem, and people have been actively working long before you to help solve it). One solution that came this year was the canonical tag, which is about to expand with cross-domain support.

Another solution remains with the AP itself. By not having its own news portal, by having stories that can disappear after 30 days, it constantly shoots itself in the foot to gain the links that would let it naturally rank better in Google.

Consider an AP story from October published with the AP’s cooperation at Google News, covering how the AP itself might want to charge some portals for early content access. Techmeme featured the story, as did several blogs like ReadWriteWeb and Mashable. By try to read the story they all linked to, and it’s gone.

For more on the issues addressed above, see these articles:

• What The Associated Press is saying to Google, Microsoft, and Yahoo
• Sorry, Tom Curley: Don’t Expect A Google Ranking Boost For The AP
• Hey AP! How About Running A Real News Web Site?,
• How The AP Fails To Get Search & SEO (Again)

But Duplicates Still Get Through

Back to the existing Google-AP deal. There, the de-duplication aspect hasn’t panned out as well as promised. It’s still possible to search on Google and encounter the same AP story being hosted by different newspapers. In addition, the theory was that if there was an AP story, it was the AP story hosted at Google that was supposed to get to billing, not the same story at members. Cohen acknowledged there are still issues:

We’re still indexing everything and showing the duplicates, but trying in the default results to show the canonical page [the AP story on Google itself]

Continuing, he explained there are challenges, in that what may seem like the same AP story in different papers might not be the exact same story.

“If they [an AP member publication] edit it or add original quotes, it begins to change,” he said. “There’s a gray area of trying to get that right and capture changes. We want to capture [and show] substantial changes but not have someone tweak a headline or byline and get listed as if it’s a different story.”

Still, it’s been over two years since Google began offering hosted wire stories. You’d expect these problems to be sorted out by now.

Not Said: Deals Stop Lawsuits

What the deals have been most successful at, at least with the AP and AFP, have been to defuse lawsuits. The AFP did sue Google; the AP threatened. Google said what it was doing with news stories fell under fair use and that the deals it cut were specifically for “new” and “extensive” uses of wire content that went beyond fair use (see AFP & Google Settle Over Google News Copyright Case for more).

Cohen reiterated this when I asked specifically about it during our interview. Google completely disagrees that listing a headline and summary of a story, with a link to that story, violates copyright laws as some publishers have contended. Nevertheless, the agreements got some wire services off Google’s back.

The AP Pushes At Google

The fires are being stoked again, however. The AP agreement is being renegotiated, and the organization has sent conflicting messages about how and when it considers listing articles to go beyond fair use.

The AP is also launching a “news registry” it says will allow it to track usages of AP content in part as a way ensure content is properly licensed (according to the AP’s view; others may disagree). Confusingly, on Friday, there came a report from Business Insider that the AP may want Google to maintain this registry (Google said they had no comment about this).

Meanwhile, an earlier project that the AP backed — the aforementioned ACAP — moves along with its own system of automatically transmitting licensing information, not that anyone is currently trying to actually license material in the way that ACAP-backers hope. Also on Friday, there came a report that from TechCrunch that Microsoft might try to woo publishers into blocking Google and get behind ACAP.

I’d previously talked with Google’s Schmidt about the business dealings with the AP, so see my interview with him for more on that (Google CEO Eric Schmidt On Newspapers & Journalism). With Cohen, I focused on more technical aspects.

Robots.txt Works Fine; ACAP Needs Progress

In particular, Google’s primary way of dealing with publishers remains the Robots Exclusion Protocol (REP). With the AP having pushed two alternatives to this, is there a problem with REP? Will Google get behind ACAP?

The AP stuff [the news registry] is still so vague that I can’t talk too much about it. The ACAP thing, there’s a challenge with some of the specific protocols. A good chunk of it can already be done with robots [REP], and it already works.

Continuing, Cohen explained further:

That’s not to say there isn’t more than can be done [with REP]. To keep that moving forward makes sense. Some things they [ACAP] want to do, it’s not a question of being bad for Google but bad for the web that opens the door to a lot of spam. For example, a directive of what the snippet should be, that this is what you have to show. One guy said the whole spam thing is overstated.

You have to think about any protocol to work for the common web, not just news publishers. We’ve had these discussions. Unfortunately, we haven’t seen much progress there.

The lack of considering the “common web” resonates with me. As my Search Engines, Permissions & Moving Forward In Copyright Battles article explains, a weakness to me with ACAP has long been that it was designed by news publishers, for news publishers while search engines deal with more content than that. From my article, I wrote;

A new system to be developed with the search engines and a broad range of publishers for online indexing. That’s not ACAP, in the sense that ACAP had not specific solutions when it rolled out. Moreover, ACAP really represents the interests of a minority of publishers on the web, news publishers. Web publishers are online merchants and small bloggers and forum owners and those with personal home pages and B2B business and Fortune 1000 sites and local merchants with single pages and more. No, every constituency can’t be represented. But any new system needs more broad-based participation.

ACAP recently rolled out an updated specification (PDF); I’ll be looking at the system and how it weighs up against REP in a future article.

Don’t Blame Us; Blame The Internet

During the interview, I also remarked how I find it personally odd to see publishers upset that Google and other “aggregators” are supposedly taking away their visitors. For me and for others, these places are our newspapers. And while publishers might prefer we start our day with them, it seems unlikely for many that this will change. But it also doesn’t have to be a negative because these same aggregators accused of robbing visitors to me also seem to be exposing news content that many would have never seen before.

Cohen commented similar to what Schmidt has said before in my interview with him, that Google gets blamed for disruption rather than changes the internet itself as a new communications medium has caused.

Eric touched on this, in how Google is often seen as synonymous with the internet. And so anything good that happens, in the space, Google did it, Google did something, and we may have had anything nothing to do with whatever innovation was out there. And anything bad that happens is Google’s fault.

So this idea, the issues that publishers have around Google News. I mean if Google News didn’t exist or even broader, if Google didn’t exist, it’s not going to change the basic disruption that’s taken place within digital media. And in fact, if anything — and I certainly see the disruption that’s there and I can recognize that — but I think you also have to recognize that the train left the station before Google came to fruition. And Google is just a tool to help you navigate that. And the different properties are there that help you find that information in an increasingly fragmented space.

NOTE: Google’s News Experiments & The Quest To Solve The “Read State” Issue is the second part of this interview that deals with Google’s experimentation with news products; a third part will be out in late November that looks more deeply into how Google News determines what to show visitors.

As expected, a revised Google Book Settlement has been filed today — about as late as possible. The agreement narrows the scope to the US, UK, Canada and Australia. It alters how revenue generated by “unclaimed works” will be handled. It formally grants retailers who license out-of-print books covered by the settlement — including Google competitors — a 37% share of sales. It also clarifies how the book pricing algorithm will work.

Google has a blog post about up about the news, which in turn points to:

• The revised settlement agreement
• Summary of changes
• FAQ page

I haven’t had a chance to go through the actual agreement, and given that it’s 10PM Pacific time, I won’t be getting through it this evening for this article. Instead, I’ll focus on what’s in the summary document. I’ll try to highlight anything I think wasn’t addressed by the Department Of Justice objections to the last settlement agreement. I’ll do this all incorporated into my live blogging of the press conference held about the settlement.

The Press Conference

On the call were Richard Sarnoff, chairman of the American Association of Publishers; Paul Aiken, executive director of the Authors Guild and Daniel Clancy, engineering director for Google Books. The first two represent the plaintiffs that sued Google. Missed the call and really want to hear? Call (719) 457-0820 or (888) 203-1112 and enter code 3915040.

Engineering Director Richard Sarnoff, Authors Guild Executive Director Paul Aiken and me for a public conference call at 9:15 PM Pacific/12:15 AM Eastern to discuss our amended agreement. To participate, ask for the “Amended Google Books Settlement Conference Call,” and use the following numbers:

Sarnoff spoke about the international scope of the settlement, and how it was changed (note, anything I exact quote should be an exact quote, as best I caught it):

“This settlement is NOT about setting up the digital future of publishing. This settlement is about not leaving ?whole? books behind …. this is about reclaiming the past. We recognized some foreign countries and rights holders were concerned …. so we’re adjusting and scaling back the scope.

OK, to better explain what Sarnoff is talking about, let’s look at what the settlement summary says:

As revised, the settlement will only include books that were either registered with the U.S. Copyright Office or published in the U.K., Australia, or Canada. After hearing feedback from foreign rightsholders, the plaintiffs decided to narrow the class to include only these countries, which share a common legal heritage and similar book industry practices. British, Australian, and Canadian rightsholders are joining the case as named plaintiffs and will also be represented on the Board of the Book Rights Registry.

In addition, as we have stated previously, we have clarified the wording in the agreement to make it clear that works that are for sale as new internationally are considered commercially available and thus Google will not display any of their content by default.

This is in reaction to part of what the DOJ was concerned about:

The Proposed Settlement operates to sweep in untold numbers of foreign works, whose authors, under current law, are not required to register in the same manner as U.S. rightsholders.  Many of those authors have never published works in the United States and are not members of the Authors Guild or the Association of American Publishers, which exclude many foreign copyright owners from membership by virtue of their membership criteria….

As the filings of France and Germany make clear, some of the United States’ trading partners have serious concerns about application of the Proposed Settlement to foreign authors and, in any event, the parties have not demonstrated that the class included representation sufficient to protect the interests of these foreign rightsholders.

Sarnoff also stressed that only those in the US can preview or by in copyright works.

Aiken spoke next, addressing unclaimed works. He summarized how funds will be be held for 10 years, not 5. How unclaimed funds will go to charities as determined by court order after 10 years. After 5 years, 25% of unclaimed funds may be earmarked for the purpose of finding rights holders.

Let’s go again to the settlement summary:

The amended settlement agreement requires the Book Rights Registry to search for rightsholders who have not yet come forward and to hold revenue on their behalf. The settlement now also specifies that a portion of the revenue generated from unclaimed works may, after five years, be used to locate rightsholders, but will no longer be used for the Registry’s general operations or redistributed to other rightsholders. The Registry may ask the court after 10 years to distribute these funds to nonprofits benefiting rightsholders and the reading public, and may provide abandoned funds to the appropriate government authority in compliance with state property laws. The Registry will now also include a Court-approved fiduciary who will represent rightsholders of unclaimed books, act to protect their interests, and license their works to third parties, to the extent permitted by law.

Just as with the original agreement, nothing in the amended settlement limits anyone’s ability to use unclaimed works. In terms of the small subset of unclaimed works that some have referred to as “orphans,” as we’ve said repeatedly, the settlement agreement takes one important step towards opening up access to unclaimed books. In the meantime, we continue to encourage legislation that provides meaningful avenues for any entity to use these works.

As for the DOJ’s view, hang on. I’ll get back to that during question time.

Sarnoff talked about competitive issues in general, and going to the settlement summary on that is actually more useful:

Syndication of All Works in the Settlement to Others, Including Google’s Competitors

As Google first announced in September 2009, any book retailer — Amazon, Barnes & Noble, local bookstores, or other retailers — will be able to sell consumers online access to the out-of-print books covered by the settlement, including unclaimed books. Rightsholders will still receive 63% of the revenue, while retailers will keep the majority of the remaining 37%. This provision has been explicitly written into the revised agreement as a Google obligation.

The amended settlement clarifies how Google’s algorithm will work to price books competitively. The algorithm used to establish consumer purchase prices will simulate the prices in a competitive market, and prices for books will be established independently of each other. The agreement also stipulates that the Registry cannot share pricing information with anyone but the book’s rightsholder.

In addition, the amended settlement removes the non-discrimination clause (commonly called the “Most Favored Nation” clause) that pertained to the Registry licensing of unclaimed works. The Registry is free to license to other parties without ever extending the same terms to Google.

This seems designed to address the DOJ’s concern that Google would have the right to sell out-of-print books because it’s covered by the agreement, while competitors had been assured outside the agreement they’d have rights to. Now, they get included within the agreement with a guaranteed revenue share. The DOJ also had concerns about price fixing, which apparently an algorithm that will “simulate” the real world is designed to rectify. Somehow, I don’t think a black box algorithm is that reassuring. But supposedly, the details of this algorithm are also being revealed.

Clancy then spoke, saying essentially Google’s all excited. This opens up more books for the world. Google’s looking forward to going beyond countries covered in the agreement and working with international rights holders.

While he didn’t address this, let me bring in the remaining part of the settlement summary:

The amended settlement does not change the primary access models outlined in the original agreement, including enabling readers to preview and purchase books, selling institutional subscriptions to the whole database, and giving libraries free access at designated terminals. Under the revised agreement, possible additional access models to which Google and the Registry might agree in the future have been reduced and are now limited to: print-on-demand, file download, and consumer subscription. The amended agreement also enables the Registry to increase the number of terminals at a public library building, and it clarifies that rightsholders can choose to make their books available for free or allow re-use under Creative Commons or other licenses. Rightsholders can also choose to modify or remove restrictions placed on Google’s display of their books, such as limits on the number of pages that users can print.

Suffice to say, the DOJ has a number of objections on access that these changes are designed to address.

Questions!

The first was from the Associated Press, who had spoken to the Open Book Alliance (which opposed the original settlment) and got a statement that the settlement doesn’t go far enough, it’s sleight of hand, surgical nip and tuck. And now the statement is live, so let’s see it in full:

Our initial review of the new proposal tells us that Google and its partners are performing a sleight of hand; fundamentally, this settlement remains a set-piece designed to serve the private commercial interests of Google and its partners.  None of the proposed changes appear to address the fundamental flaws illuminated by the Department of Justice and other critics that impact public interest.  By performing surgical nip and tuck, Google, the AAP, and the AG are attempting to distract people from their continued efforts to establish a monopoly over digital content access and distribution; usurp Congress’s role in setting copyright policy; lock writers into their unsought registry, stripping them of their individual contract rights; put library budgets and patron privacy at risk; and establish a dangerous precedent by abusing the class action process

So the response to that? Clancy stepped up, saying there were lots of discussions on how to change things. Adjustments were made to address class member concerns (the people involved in the lawsuit, rather the the Open Book Alliance, which is not a party to the suit). “I understand Amazon, Microsoft and the Internet Archive don’t want to increase access to these books,” he said, or very close to that. That was a zinger, stressing that the Open Book Alliance just happens to be backed by major Google competitors. Not that Google minds. Clancy said they welcome the competition and feel the settlement addresses concerns.

Aiken: “These are substantial changes.” He added that yes, the core settlement was largely protected but that it had to be, as it was in general seen correct.

Sarnoff: Said he assumed the OBA hadn’t read the settlement. That was probably true enough. The press conference itself appears to have started about 1/2 hour after the settlement was out. Some reporters on the call mentioned they hadn’t even read it.

The next question was if there had been feedback from the Justice Department about the changes.

Sarnoff: Had discussions with them and feel have addressed their key concerns.

There was another question I then largely missed. I put in the next one. After scanning the summary, and the DOJ objections, I felt these two objections were largely not addressed:

Because the owners of orphan works are an incredibly diverse group that includes not only living authors or active publishers, but heirs, assignees, creditors, and others …. these rightsholders are difficult or impossible to locate, and thus difficult to notify.  Moreover, no amount of notice is likely to protect those orphan rightsholders who are unaware of their rights or unclear how or whether they want to exploit them.  Yet, if an out-of-print copyright owner does not come forward within five years, profits from the commercial use of the out-of-print work are distributed to pay the expenses of the Registry and then to the Registry’s registered rightsholders.

and:

The Proposed Settlement does not forbid the Registry from licensing these works to others.  But the Registry can only act “to the extent permitted by law.”  S.A. § 6.2(b).  And the parties have represented to the United States that they believe the Registry would lack the power and ability to license copyrighted books without the consent of the copyright owner – which consent cannot be obtained from the owners of orphan works.  If the parties are correct, the Registry will lack the ability to provide competitors with licenses that will allow them to offer to the public anything like the full set of books Google can offer if the Settlement Proposal is approved.

In short, there’s a big issue with the so-called orphan books. The owners of these books don’t give permission to be included, and Google gets a license under the settlement to use the works anyway. Why not exclude them from the settlement and defuse a major issue?

Aiken: We don’t know which ones are the orphan works. There’s lots of confusion of out-of-print works with orphan works. The success rate of finding out of print authors is quite good. Once you have some money for to locate them, the success rate in finding authors is around 90%.

Clancy: The opt-in model is the model that Google uses today. The problem that we were solving here is not the problem of orphan works. It’s the unclear rights situation. If you did the opt-in model …. then you don’t have an organization that can effectively …. part of the way you’re going to find them, is that you have money being held for them. For us, we felt …. this will be a database where books are claimed …. we don’t believe that will happen today [without a settlement nor through congressional action]  …. we don’t think [leaving out the unclaimed works] would solve the problem and find all these works that aren’t [really] orphans at all. We don’t think an opt-in solution would open up many of them.

This confused me. By their nature of being unclaimed works, the authors and publishers never opted-in to being scanned. So how can Google say the model is opt-in?

Aiken explained that Google’s general model for adding books not covered by the settlement is opt-in. Clancy then also clarified that for all known works [that are still in copyright], opt-in is used. And that the actual number of orphan works is small.

In other words, for books in copyright with known authors or publishers, Google asks permission for them to be included through its partners program. But the library scanning project, that has some books that were scanned which are in copyright but without known rightsholders. While these were scanned on an opt-out basis, the settlement effectively converts them into opt-in by holding that permission (and any revenues) in trust, for when the rightsholders are found.

Which, frankly, still isn’t opt-in. And given that everyone is so positive that you CAN find rightsholders for most of these unclaimed works, why not go out and find them first, then ask if they want to be included. Surely the settlement can generate enough money from books with known authors to fund that without having to include these books at the outset?

Aiken: Underlying all this is a mess that only a class action can clear up. There are books that had no digital rights ever granted. To sort through all those would be a burden. You would not solve this problem. You can’t solve this.

I did a mental double-take on this. Seriously, the only way to solve the rights problems of books with unknown works is through a class action lawsuit that started over whether scanning books was a violation of fair use? An issue itself that isn’t even being resolved by this but just side-stepped. Why not ask the US Congress to pass laws to address this?

Aiken responded to say that this is “our solution, a solution using class action law” and stressed that class suites are a part of civil procedures authorized by Congress. Somehow, I don’t think that’s going to satisfy some of those who feel the settlement goes to far into establishing new copyright law, a concern even the DOJ raised.

Clancy added that on orphan works [real ones, those without authors that can be foudn], they don’t think legislation will take decades:

In fact, what we think what this settlement does is solve one of the biggest challenges with orphan works …. that no database base exists …. it will be made public [when the registry established by the settlement locates rightholders of unknown works, anyone can tap into that database to contact them] …. we think that helps everybody, because everyone can turn to it. We can get good orphan work legistlation done.

Sarnoff: We are certainly hoping that this settlement is the key that unlocks a positive outcome on the legislative process on orphan works as now there’s a way to actually implement any legislation that congress decides on orphan work [IE, there's a registry for rightholders that can tie into legislation] …. thinks much confusion that orphan works are not the same as out-of-print works. Vast amount of out-of-print works that will be claimed …. so we have three diff buckets. One large bucket of out-of-print works, the vast majority will be claimed. Orphan works that will be claimed when revenue is attached to them. Lastly, whatever the remainder is that never get claimed and where rights holder can’t be identified, those are the orphan books where Congress may want to take action.

There was one more question on institutional sales that got beyond me, then the call ended.

Over the coming week, there’s going to be a lot of commentary and analysis of the revised proposed settlement. We’ll revisit it here, ourselves. Until then, a few resources:

Department Of Justice Files Objections To Google Book Search Settlement: Our summary of the DOJ objections to the original settlement, from September.

Google’s Schmidt To Book Settlement Critics: What’s Your Solution?: Also from September, this provides a brief overview of the case in general and the orignal settlement.

The Laboratorium: James Grimmelman is a lawyer who has been following the case shortly. He doesn’t have an analysis up yet, but he will. If you can’t wait, look at his tweets for today. He does a change-by-change summary, tweet-by-tweet.

Press Review+: Google Book Search Revised Settlement Settlement (2.0) Released; What About Libraries?: From ResourceShelf, a growing resource list of coverage.

Also see related discussion at Techmeme for even more press coverage and commentary.

Yet another News Corporation executive is talking about Google, and yet again, I feel like they have no concept about how Google interacts with their web pages. Which is frightening, since they’re being very vocal about how they’re supposedly wronged by Google. Please, someone, give them a search marketing 101 course.

In the latest volley, News Corp’s president and COO Chase Carey tells Business Insider that being in Google is a “backdoor” that doesn’t “make sense” to offer if they’re trying to sell paid subscription.

Sigh.

Yes, you can use Google News as a way to get around the Wall Street Journal’s paywall. Through the First Click Free program that Google offers, anyone coming from Google has to be admitted to see the actual article. In fact, I’m a guilty party in telling people how to do this (see my Read The Wall Street Journal For Free post).

Here’s the thing. I’ve never, ever, heard anyone at the WSJ or News Corp before suggest that this is somehow a major problem, that this is a backdoor that’s being massively abused. To me, it’s similar to how there’s illegal music on the web. Sure there is, but for many people, it’s easier and cheaper just to buy a song legitimately. I suspect that many visiting the Wall Street Journal from Google News are:

• Interested in a one time article, will never buy a subscription, so you’ve lost nothing but gained an ad view (which are still valuable)
• Are not tying to “hack” an entire daily reading experience
• A few decide to buy a subscription (hurray, you made money)
• Some, like me, already have a subscription (and you’ve lost nothing)

Here’s the more important thing:

FIRST CLICK FREE IS NOT REQUIRED. YOU DON’T HAVE TO DO IT, AND YET YOU CAN STILL BE IN GOOGLE.

Sorry about all those caps and bolding, but after reading so much commentary earlier this week about News Corp’s Rupert Murdoch potentially taking his news content out of Google, I was amazed that this key point was being missed.

Publishers can have Google News index the entire text of their articles but NOT show the full story to visitors who come from Google (for Google’s web search, that’s not an option — but you can provide summary pages). They can also, if they choose, have only a small summary of their content indexed. These aren’t special hidden features that News Corp wouldn’t know about. Both are explained in Google’s public help files.

For example, here’s how it works for Newsday:

See that first story I’m pointing at, in response to a search for long island flooding? It’s from Newsday. But if I try to read the story:

No luck. I have to subscribe. No First Click Free, no passing Go, you collect your $200.

By the way, to my understanding, News Corp’s Barron’s has done this in the past and may still do this, so News Corp really should be aware of the option. Also, Google is supposed to flag this type of content with a “subscription” label, but that’s not being done for Newsday, probably due to a glitch. I’m checking on that.

So if Carey doesn’t want a backdoor, he doesn’t have to have one. He can shut it himself. Moreover, he can shut it and yet not have to shut the door on Google entirely. That’s important for those like Mark Cuban who like Murdoch’s “We’d rather have fewer people coming to our web sites, but paying” view, as Cuban commented to me in a discussion on Mathew Ingram blog post about the value of visitors. Cuban said:

The prevailing wisdom seems to be if you build it and they come, why not let them in the door ? The answer is because they get there, but don’t really know or care where they are at. They went to google and got what they were looking for from some site google sent them to. I know the numbers i have seen from news sites show that retention of uniques earned through google searches is not high. They are visitors. They leave. There is no relationship developed.

IMHO, just as people ignore the banner ads on pages, they ignore the name of the company presenting the content. One is just above the other.

You are the search expert, but my business sense tells me that its better to focus on the people who are there to dance with me. Block google. See what the information tells me. See what happens. I dont think you lose anything, and you might just hit a home run.

Let’s assume most people from Google don’t convert. So what? Block First Click Free, and you still get traffic from Google. You provide no “backdoor” access; you lose nothing. You only talk about Google as a backdoor that harms you if:

1. You don’t fully understand how Google indexes you
2. You have a deliberate agenda to put out false information

You also don’t say things like how you don’t need or want traffic from Google to News Corp publications when you’ve got parts of your organization actually spending your cold hard cash to buy that exact traffic. Here’s an ad for the Wall Street Journal on Google for financial news:

By the way, Carey didn’t just say that Google News readers should be cut-off from seeing free articles. He said he didn’t want people using any “channels” from getting there. So I guess that means the much discussed deal with Digg back in 2007 is on the chopping block, too. You know, the one that got a big splash because if someone submitted a story to Digg from the WSJ, then anyone could click from Digg and read that story?

Also, if the Journal really wants to be consistent, then you need to stop with the giveaway of ANYTHING on the site. As my Read The WSJ Free post also explains, the WSJ site itself has free content that it puts out for anyone independently of Google News. In fact, at Nieman Journalism Lab, WSJ executive editor Alan Murray explicitly described how the WSJ drops the paywall independently of Google, Digg or whatever — how you can start right at the WSJ site itself and get plenty of free stuff.

Rupert Murdoch added the Wall Street Journal and other Dow Jones properties to News Corp over two years ago. I understand that it takes time to fully digest what he purchased.

I understand he and his executives might not realize all the various ways that they themselves might be inconsistently fueling the same news “promiscuity” that they’re pinning on Google (see Dear WSJ: To Avoid Google Disease, Please Put A Condom On Your Content).

I understand Murdoch and his executives may not still realize yet how News Corp does some of the same evil aggregation that it rants against (see A Look At All The Sites Owned By Rupert Murdoch That ‘Steal’ Content from TechDirt for more).

But still, it doesn’t make much sense. I don’t understand why he is attacking Google so hard when he has so much control over his own content. He can keep them entirely out. He can let them send visitors but not give anything away at all. So could any newspaper, right now, today. But his executives have positioned Google as a “vampire” that’s somehow caused all this disruption.

Pondering this more in my head on Tuesday, it occurred to me. Rupert Murdoch doesn’t have a very good reputation in some news circles. That from the same article in the Wall Street Journal announcing his purchase of that publication, we had this:

Opponents of the deal called it a dark day for journalism. Leslie Hill, a family member who opposed the deal, resigned as a Dow Jones director late Tuesday afternoon. In a letter to the board, she conceded the deal was a good one in financial terms, but said it failed to outweigh “the loss of an independent global news organization with unmatched credibility and integrity.”

If you’re seen as a threat to quality journalism, as someone who oversees another part of your empire (Fox News) that the White House is trying to dismiss as a news organization, what better way to be seen as a journalism champion than to speak out as if you’re protecting all journalism from the evil of Google? Build Google up as the real threat to journalism. Suddenly, other media execs who viewed you as the “bad boy” of the journalism world are now looking at you with newfound respect (despite the fact that most of them are not “must carry” and won’t get any potential deal you might land).

Now THAT makes sense. Alternatively, it just makes sense that the News Corp execs simply don’t get how their businesses operate online, in respect to Google. Neither prospect is encouraging.

Postscript: The Daily Telegraph has remarks from Jonathan Miller, News Corp’s chief digital officer, saying that his company would block Google within “months and quarters — not weeks. He also wrote said News Corp could survive without the traffic and, in fact, that it didn’t have much value:

“The traffic which comes in from Google brings a consumer who more often than not read one article and then leaves the site. That is the least valuable of traffic to us… the economic impact [of not having content indexed by Google] is not as great as you might think. You can survive without it.”

As you’ll see in the comments below, I’m sure they and others can survive without it. But I suspect they could thrive with it. What’s amazing is that as shown above, News Corp spends money on ads at Google to attract the same audience from Google that they say isn’t as valuable. So we’re either getting PR spin or they’re doing a poor job of trying to convert those Google visitors.

With the ads, they can tell someone is after business news in general and send them to a very targeted landing page. With an click to a news story, someone’s intent IS different. They aren’t explicitly looking for site about business news, for example. But they clearly are interested in business news (or some type of news, as they’ve come). If they’re not converting, part of the issue is probably that the Wall Street Journal has done a poor job of trying to convert them. Rather than show them the same layout as you’d get when coming into the site, potentially they could have a much more simple display — with much more of a tailored pitch for getting more news.

Via Business Insider, there’s also a video of Miller’s remarks, though there are some weird audio issues with it:

With Rupert Murdoch talking about blocking Google, Jason Calacanis suggests that Murdoch should cut an exclusive deal with Bing. If that doesn’t kill Google, it might at least win Bing a 10% search share or more within 6 months, he argues. I suggest Jason needs a reality check. Here’s one for him.

From his post today, Jason writes:

I put forth a simple strategy for Microsoft to pursue with Bing in which they would go to content providers like the New York Times or Wall Street Journal and offer them 50% more revenue then they are currently getting from Google search referrals to be exclusively indexed in Bing….

So, for a moment, imagine a world where Bing could say in their TV commercials:

“Want to search the New York Times, Wall Street Journal, USA Today and 3,894 other newspapers and magazine?”

“Well, then don’t go to Google because they don’t have them!”

“Go to Bing, home of quality content you can trust!”

OK, it’s a busy day for me, so I’m going to ramble a lot of off-the-cuff stuff that still ought to toss a big bucket of cold water on Jason’s high hopes. And I look forward to comments from anyone who wants to pull harder numbers together for some of the stuff I’m mentioning.

First, the AP has already dropped hints recently it wants to do the same thing, at least give one of the major search players an advance on stories. Maybe it can sell Bing on that idea. We’ll see. Google’s not blinking so far. If anything, Google CEO Eric Schmidt suggested last month that the AP has an inflated view of how valuable it is. From my live-blogging of his remarks in a meeting with reporters:

I want to distinguish betwen what you think your content is worth and what it’s worth [laughs all around]. We’re not going to use the price you think as a signal in the results.

Now let’s take your 50% more on search referrals idea. So Bing will offer to pay the Wall Street Journal 50% more than the ZERO Google currently pays for millions of referrals it sends the Wall Street Journal and other newspapers each day right now?

These newspapers don’t pay to be in Google’s editorial listings. They get tons of traffic from Google for free. What you really mean is that perhaps they could hope to convince Bing to pay them if they block Google from having their stories (and really, a headline link and short summary of their stories, not the stories themselves) in return for a gamble that Bing will send them either as much traffic or less traffic but more revenue.

OK, Jason, what’s that check that Bing needs to write to the WSJ? I don’t know how much traffic the WSJ gets per day from Google — for free. But we can take Compete for a starting point. It says the WSJ gets 12 million visitors per month, with 11% of those coming from Google –  about 1 million visitors per month. That feels low to me, but it’s in line with what some UK papers get.

Bing’s got about 10% of the search share; Google 70%. That’s from usage stats, by the way, not from stats based on actual traffic sent to sites. In that, Google’s much higher. Mahalo has stats — bet you’re in the 80% of all search traffic range from Google, aren’t you?

So right now, Bing’s about 1/7th the size of Google, if I’m doing my math right. Murdoch could do his deal with Bing as you suggest and thus lose his 1 million visitors in exchange for just over 150,000.

OK, so what’s the value of those remaining 850,000 million visitors? Remember, Murdoch’s also talking that he doesn’t mind a smaller audience if it’s one that pays. But those 150,000 million from Bing aren’t going to be paying. They’ll be just like people who come from Google now, needing to get free access to the Wall Street Journal at least for the first story they click through from or otherwise becoming very annoyed at Bing for sending them smack into a paywall.

Also, it’s not really an 850,000 visitor difference we’re talking about. I mean, Bing exists right now. It should already be sending 150,000 people to the Wall Street Journal, if we work off a direct proportion to what we guess Google already sends now (and that’s very rough work). Really, doing the Bing deal as you suggest doesn’t give the WSJ a net gain. It just potentially cuts off those 1 million people from Google.

So what are those people worth? What’s the amount Bing needs to pay to make up for all that lost traffic? Are they worth $0.50 per day – (or $15 per month; $180 per year)? That’s about $180 million per year Bing needs to pay just to the WSJ alone to guarantee they’re not losing that $0.50 per day, per person, that they might be earning. Maybe they earn a lot less; quite possibly, they earn more.

And that’s just to pay the WSJ, right? What about all the other papers that want to put their hands out? Does the New York Times get the same deal? It’s not hard to get up to 1/2 billion in money to license the right to link people to content on the news providers web sites, all in the hopes that this will make your search engine overall THAT much more compelling that Google overall.

No, I don’t think that makes sense for Bing todo. Keep in mind that most people at Google do NOT go to Google News. Stats from last year from Hitwise put Google News at 1% of Google’s overall traffic. Also go back and look again at those Compete stats for the WSJ. Bing’s portal partner MSN and Yahoo both send the WSJ more traffic already than Google. There’s almost certainly a licensing deal in place, and it clearly doesn’t seem to be generating enough revenue to make Murdoch declare that the war for payment in online is over.

Maybe Bing would see some growth, which would send it more traffic, which would mean the minimum revenues it needs to pay the member papers would be less. But I doubt it would be much.

Moreover, there are paid subscription services that you can tap into right now that give you the ability to search and read content from a variety of newspapers and magazines. They clearly aren’t that compelling to the majority of searchers out there.

Also, how much of the WSJ is vital information that absolutely cannot be found in other sources? Even the WSJ knows that much of what it has isn’t unique. They do have great content — I know, I actually take the print version and read it usually each day. But on a day I skip it, I’m hardly out of the loop newswise.

So you ask in your post, What’s the percentage chance I think Bing will do as you suggest? About 1%.

The reality is that the Wall Street Journal execs seem to be leading a charge against Google without really knowing where they are going or what they want. They already get lots of traffic plus get to have a paywall, thanks to First Click Free at Google. And yet, Murdoch doesn’t seem to know exactly how that operates or how Google indexes his paper. His managing editor Robert Thomson gets confused about font sizes and how Google News works. I get the impression both of them are good at talking but don’t know the actual realities of their traffic situation in relation to Google. Or they know it well but are happy to ignore it.

So why are they talking so much? At best I can tell, they seem to want a licensing deal similar to what the AP, the AFP and a few other services get. Perhaps they’ll even get it, and if so, I imagine it would be on the order of 10s of millions, not hundreds of millions. And the smaller papers out there won’t get it, as I’ve written before (see Garlic For The Google Vampire). They’re not “must carry” publications. Murdoch may sound like he’s speaking for the industry but really, he seems to be speaking to strike his own particular deal.

But hey, let’s assume they do a deal in the end. Let’s say the AP does one, too, where they withhold information from Google completely or for a set period of time. Are both publications going to stop using Google? No. And if they’re not, who’s going to start talking about how hypocritical they are? It’s fine for them both to have literally hundreds of reporters each day using Google to mine information for their web sources, something that’s free for them to do? (See Do Newspapers Owe Google “Fair Share” Fees For Researching Stories for more on this). Or are they going to start paying research fees to Google, so it can in turn redistribute those fees out to the millions of resources it currently carries — upon which they currently depend?

Finally, let me end on that trust thing. I know that lots of people who post and blog on the web have a growing mistrust of Google. But ordinary people still seem to trust it, a lot. Survey after survey shows it as a top brand. And in a time when search results between the major search engines aren’t that different, what’s the key element Google has over its rivals?

It’s trusted. It is a trusted friend that diligently serves millions who use it each day. Your ad campaign of how Bing could say it’s the home of “quality content?” People already think they’ve got that at Google. I don’t see it flying.

Postscript: From Hitwise, some statistics. According to them, both Google overall and Google News send more traffic to the Wall Street Journal than any other site, 25% combined.

Postscript 2 (Nov. 22, 2009): Since this was written, we’ve had more news that News Corp is being enticed by Bing. See:

• Badda Bing! Microsoft woos newspapers by funding their stick to beat Google
• Microsoft and News Corp eye web pact

The first is from TechCrunch about a week ago; the second from the Financial Times today. And before things get spinning that this was all Jason Calacanis’s idea — while I like Jason, he used to work for Jonathan Miller, who heads digital for the Wall Street Journal. Suffice to say, I think News Corp is putting out a lot of trial balloons — I suspect Miller mentioned this idea to Jason, who in turn put it out there. I highly doubt in the space of two weeks that News Corp said “get me Microsoft; this sounds like a plan. Or vice-versa.

I suspect that News Corp’s most valuable content of all its properties remains the Wall Street Journal. Google operated just fine for years when the WSJ wasn’t included. I highly doubt this would hurt them. That’s especially so because, as even the WSJ has acknowledged in the past, people can locate other sources for the news they report.

Also, it’s noteworthy that virtually all major news publications in Belgium a few years ago opted-out of Google (absurdly suing to get out, rather than using voluntary systems). The lawsuit was more about hopes to blackmail Google into paying the papers to be included. Instead, Google conceded and dropped them. And down the line, the papers effectively came crawling back asking to be reincluded. Belgian Papers Back In Google; Begin Using Standards For Blocking has more. Whether News Corp’s content is more valuable to Google than that of an entire country’s remains to be seen.

Finally, it’s an extremely odd move to me for Microsoft to be trying to strike exclusive deals like this. It’s one thing to license content. It’s another to try and suggest that a competitor be locked out. Microsoft has a terrible anti-competitive reputation. It’s also trying to convince regulators in the US and Europe that it should be allowed to acquire Yahoo’s search technology, because that will — it argues — make a better competitor to Google. You can expect that Google will use an anti-competitive partnership with News Corp as a sign of arguing that Microsoft is “back to its old tricks” to pressure against such a deal.

See also these past posts, that provide more background and perspective:

• Would Someone Please Explain To News Corp How Google Works?
• Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers
• Google’s News Experiments & The Quest To Solve The “Read State” Issue

The long-running debate over Google and its impact on newspapers and journalism took another turn today when News Corp founder Rupert Murdoch said his company may makes its sites invisible to Google, and Google fired back by saying, in essence, bring it on.

It began with this interview on Australia’s Sky News (which Murdoch owns), reported by the Australian site mUmBRELLA, in which Murdoch reiterated his belief that Google and other search engines “steal” their stories:

“The people who just simply pick up everything and run with it, and steal our stories. We say they steal our stories — they just take them without payment. There’s Google, there’s Microsoft, Ask.com … there’s a whole lot of people.”

Murdoch agrees that Google sends his news sites a lot of traffic it might not have gotten on its own, but questions the value of that traffic to advertisers. He says, plainly: “We’d rather have fewer people coming to our web sites, but paying.” And when asked why his sites haven’t made themselves invisible to Google and other search engines/news aggregators, Murdoch says: “Well, I think we will. But that’s when we start charging.”

To that, Google fired back today, telling the Telegraph that, essentially, they don’t care if Murdoch wants to block its sites from being found via search and/or Google News.

A spokesman for the search giant said: “Google News and web search are a tremendous source of promotion for news organisations, sending them about 100,000 clicks every minute.

“Publishers put their content on the web because they want it to be found, so very few choose not to include their material in Google News and web search. But if they tell us not to include it, we don’t.”

Google is referring, of course, to using robots.txt or a similar protocol to keep content from being indexed. Danny Sullivan wondered aloud why Google’s critics in journalism weren’t already doing that, especially after the Wall Street Journal recently accused Google of “encouraging promiscuity” online by allowing searchers to bounce from one news site to the next with no loyalty. Danny also sat down recently with Google CEO Eric Schmidt for a lengthy conversation about Google and journalism.

The debate/battle is far from over. The question now, at least where Google and Murdoch are concerned is: Who’ll blink first?

Postscript: Bill Tancer has posted about this on the Hitwise blog today, specifically with a look at the traffic that Google sends to the Wall Street Journal’s web site:

“…on a weekly basis Google and Google news are the top traffic providers for WSJ.com account for over 25% of WSJ.com’s traffic. Even more telling. According to Experian Hitwise data, over 44% of WSJ.com visitors coming from Google are ‘new’ users who haven’t visited the domain in the last 30 days.”

Note From Danny Sullivan: I’ve also posted some related thoughts today in my Why An Exclusive Wall Street Journal Deal Wouldn’t Help Bing article.

Google News has made it easier to customize its home page with the creation of a Custom Sections Directory that includes material created both by Google and by Google News users.

The directory includes sections such as “Mobile Technology,” “Google,” “Social Networking,” “Apple Inc.” and “NFL.” You can browse the directory by category (Business, Science, Sports, etc.) or search for custom sections by keyword. The directory also offers tools to create new custom sections based on zip code or keyword and add them to the public directory.

When done, the sections you’ve added will appear as links in the left-side menu and as new sections at the bottom of the Google News home page.

According to today’s announcement, this new directory is available to Google News editions in Australia, Canada, India, Ireland, New Zealand, the United Kingdom, and the United States, with more countries to be added in the future.

Associated Press president Tom Curley and News Corp CEO Rupert Murdoch both made speeches yesterday in China that are being widely reported as an attack on Google. Having read through the actual speeches, I was surprised they weren’t as bad as I thought they’d be. Below, I’ve highlighted key portions along with my own commentary. And hey Facebook, looks like AP doesn’t like you, either.

Let’s start with Curley’s speech:

President Li, I salute you for your leadership and vision in organizing this summit. The timing could not be better. The Associated Press and I are honored to join with all your guests in exchanging ideas at this moment of profound economic, technical and cultural change.

Awesome. Any salute about the press restrictions that China imposes? But I digress….

The marketplace for news content is growing. More people in more places seek out news more often than ever. Yet, we don’t get paid appropriately for our hard work and the risks we take.

OK, this is debatable. Some people have a fundamental disagreement with what the AP thinks it should be paid for. But continue…

Free-riders and pirates are claiming they’re entitled to our property. And we face challenges in adapting to a world where our former customers – consumers of news – easily can help produce or report the news. Whether you live in west Texas or west China, news can come from tweets even before agencies as AP or Xinhua have found out and begun their reporting.

And who are these people? Any distinction in the real debate between free use and outtright theft? More interesting — when agencies find out about news from news consumers, are they figuring out a way to reward the channels that made that possible (say Twitter, Facebook, Google Blog Search?)

Next we get a long lesson on how basketball rules changed to lead into the AP looking for its own “game changer.”

In the next couple years, all news organizations face the same mission – get all the way across the burning bridge from analog to digital journalism and to make the difficult choices that this crossing presents.

Because they’ve been standing on one side of that bridge for over a decade, and if it’s mostly burned through now, it’s because they were so hesitant to cross or inept when they tried. But go on…

The changes are so radical and pervasive that after nearly 15 years of doing business on the Internet, news organizations are still testing long-held assumptions about what the other side of the digital bridge actually looks like.

From all visible signs, it’s not a place where a news organization can survive by just doing business as usual, creating and marketing content. Indeed, it often appears as if the more our content is distributed, the more our pricing declines.

Sadly, these types of speeches often make you think they’re doing just that, assuming it’s business as usual.

And yet the audience on the other side of the bridge seems to be having fun. They’re doing other things besides reading news stories – like connecting with their friends on Facebook, networking through LinkedIn, writing blogs by the millions and following interesting people, places and things on Twitter.

Even before they started paying more attention to each other than to Web sites, users had adopted search as their online compass for everything including news. That easy-to-use tool led them to cultivate new sources for what they sought. Wikipedia, once a user-generated curiosity, now sits atop the leaderboard of news and information providers with a staggering monthly audience of 60 million users.

I feel kind of bad here, like the AP is mad that people aren’t reading its news in the way that a higher power apparently intended. Whatever that is, because the AP has long failed to make it possible to read AP content on its own site.

Discussion of how to change AP’s approach to digital news quickly led to the almost embarrassingly simple idea of an AP news blog. We, like others, had in the recent past looked at this approach, tried it in limited ways and moved on.

Ironically, the embarrassing simple idea probably would have generated the AP tons of traffic and people pointing at it for years in the ways that they instead point to other sources that carry AP content or cite it.

But this time we decided to join the fray in a way that could be game-changing – look at blogging as the foundation for a model of audience engagement.

Rather than work only with theory, the team decided to put a real model in play – an experiment around the week-long Sonia Sotomayor Supreme Court confirmation hearings in July.

OK, cool. Hadn’t realized AP did this…

The blog idea served as the center of gravity for the new model, and other audience channels were connected to create an experimental new “ecosystem.” Twitter became a facility for direct user comments and interaction. Links back and forth to member news providers, recruited in advance, were included to test the blog’s ability to move traffic around a network.

The experience provided a model for what a collaborative approach to a networked news environment might look like – key ingredients being a major portal as a traffic hub, social media like Twitter for audience input and member participation to create the network effect.

The combination was very successful from all perspectives. Yahoo’s news traffic jumped 20 percent compared to the norm for AP stories. Participating newspapers more than doubled their Web traffic from Yahoo. Twitter drove 27 percent more traffic to Yahoo than normal, and AP’s blog attracted triple the traffic of the regular news story. Not incidentally, we estimate the net potential new revenue generated by the effort to be at least $200,000 greater than expenses.

Well duh. I mean duh. People in all areas have been telling the AP stuff like this for years. What took so long? Welcome, welcome, AP, to the world of doing online news right.

And we content creators have been too slow to react to free exploitation of news content by third parties without input or permission.

Random distribution of traffic by aggregators such as search engines directs audience and revenue away from those who invest in original news reports but assures the aggregators and their ad networks of a stream of revenue based on aggregation and indexing of published news content.

Crowd-sourcing web services such as Wikipedia, YouTube and Facebook have become preferred consumer destinations for breaking news, displacing Web sites of traditional news publishers.

Notice Google’s not mentioned once. Just implied as part of the “search engines” mention. Notice also that Wikipedia and Facebook kind of get put on notice, neither of which I think has been named before.

To turn the tide, AP is creating a News Registry – a rights management and tracking system.

Participation in the News Registry would discourage unauthorized exploitation of news content by third parties and promote uses that benefit participating news publishers. Implementation of digital protocols that convey rules pertaining to access and use of published news content would enable publishers to pursue individual and collective licensing opportunities.

Ah, yes, the News Registry. Of which there are many, many questions about how it will really work. Questions I’d love to put to the AP, but last I tried, I was told they were done discussing it.

AP’s members have agreed to participate in a NewsMap or a constantly updated index of original news content submitted to the News Registry to steer aggregator traffic, including from search engines, to the Web sites of participating publishers, benefiting the publishers in terms of audience and revenue terms.

Yes, we’ve heard of this, also. And if the members are required to point at AP’s stories through a contractual relationship, then those potentially will be seen as paid links by Google. And if so, then they’ll have to use the nofollow attribute or risk having the AP being banned by Google for violating Google’s editorial guidelines. The AP doesn’t get to launch a massive link network just because it’s the AP.

Finally, AP is creating a NewsGuide or an aggregated body of unique news content, curated by news editors. The NewsGuide would enable news publishers together to create a preferred web destination for consumers of breaking news, which also would serve as a conduit to related content displayed on the publisher Web sites.

I haven’t heard much about this part, but it goes with the next three points summary:

We call this effort AP3P or Protect, Point and Pay.

Step one is to protect published news content against unauthorized exploitation.

Step two is to aggregate and index published news content so that aggregators can better point their users to the published content.

Step three is to enable new content licensing models for use of the published content with support for payment models that individual publishers may adopt.

So … build their own news portal, get people to point at it, still try to pressure those without agreements or quoting the material that fair use doesn’t exist.

The News Registry will use a common taxonomy and format around intellectual property rights and licensing rules. It will reflect common understanding around the aggregation and indexing of published news content and will enable participating publishers to share mutually in new licensing opportunities whether based on subscriptions or advertising.

Which sounds a lot like the ACAP system that the AP also backed a few years ago and now seems to be abandoning. Which had the same problems — that news publishers are not the only ones with licensing concerns, and a system they concoct on their own may not be adopted by others.

There also is one more “P” that’s important. Going forward, AP will license only those who agree to the principles of Protect, Point and Pay.

We will no longer tolerate the disconnect between the people who devote themselves – at great human and economic cost – to gathering the news of public interest and those who profit from it without supporting it.

Now that feels aimed at Google, which is in negotiations to renew its existing agreement with the AP. If the AP decides Google’s not playing ball using the new rules it wants to dictate on fair use, then it won’t strike a license.

And if it does that? Google will probably keep indexing web content as it has done before and let AP file an actual lawsuit as it threatened before. Maybe we’ll get some clarity in that way.

The public must have broad access to the news, and digital distribution can enhance that access. That distribution must occur in such a way that it supports newsgathering in the public interest – for those engaged in the resource-intensive work of reporting on government actions to those who risk their lives covering global conflicts, and to all the news coverage that has the potential to help people improve their lives.

I actually think few disagree with the hard work and role in life that news organizations perform. Virtually all of them are for-profit enterprises, of course. Despite that, many would like to see them survive. But the question remains — are they doing badly because of Google / the Internet / pirates / whatever or because they’re run by people who have simply failed to adapt and are now looking for someone else to blame?

On to Murdoch’s speech, which dealt far less with news content but still had some points to highlight:

Too often the conventional media response to the internet has been inchoate. A medium once thought too powerful has often seemed impotent in the past few years. Of course there should be a price paid for quality content, and yet large media organizations have been submissive in the face of the flat-earthers who insisted that all content should be free all the time.

There are plenty of people who think you should pay for content, me included. I operated a successful premium content site for years. Murdoch’s Wall St. Journal is one of the best examples of one in the mainstream media. So the point…?

There are many readers who believe that they are paying for content when they sign up with an internet service provider, presuming that they have bought a ticket to a content buffet. That misconception thrived on the silence of inarticulate institutions which were unable to challenge the fallacies and humbug of the e-establishment.

Gosh, I suppose when I bought my TV, I thought I just got all that free stuff out there. But then I couldn’t get to HBO. Turns out I had to pay for it. HBO convinced me I should seek them out, and I did.

The aggregators and the plagiarists will soon have to pay a price for the co-opting of our content. But if we do not take advantage of the current movement toward paid-for content, it will be the content creators, the people in this hall, who will pay the ultimate price and the content kleptomaniacs will triumph.

Will that also include reporters at mainstream publications who quote other publications without explicit permission, who don’t link to them or who research stories for free using tools like Google? Would it include Murdoch himself, who said later:

A cursory search of the internet will throw up some rather vigorous and vitriolic criticism of this curious character called Rupert Murdoch.

I’m guessing he did that search on Google, which while he didn’t attack by name in this speech, has clearly been one of his targets recently. He doesn’t pay for using Google. He probably felt the summaries of that criticism that Google listed was fair use. But he also seems to suggest in the same criticism that listing pages, providing summaries and linking to those pages is something Google should be paying each and every content owner for. Such a system probably goes against accepted fair use. Certainly it would be hard to implement. It neglects the fact that anyone who wants out of Google can get out with a simple electronic “do not enter” sign that’s been a standard for over 10 years.

Also see my recent article, Google CEO Eric Schmidt On Newspapers & Journalism, which touches on some of the issues above in more depth, along with Google’s views on many of them. Also see more discussion on Techmeme.

Postscript: Also see What The Associated Press is saying to Google, Microsoft, and Yahoo from Zachary Seward at the Nieman Journalism Lab. It’s a must read. In short, in post-speech remards, AP’s Curley seems to believe he’ll partner with Microsoft to ensure AP content gets top ranking in Microsoft portal and search properties (like Bing) and that Microsoft will be getting past Google with new visual search technology. Which caused me to comment:

Google could easily say thanks, no thanks, sue us if you think showing summaries and links is a violation. The hosted AP stories, to my understanding, is something the AP wanted from Google — not something that Google cares about having.

All this continues to make me think (or be scared) of just how fundamentally off the AP seems to be at the highest level. That they just don’t get how Google operates, that it cares to point out (not in).

I know Microsoft’s search technology as well if not better than Curley. If he’s drinking the koolaide that they’re going to blow Google away with some new tech, he’s in for a rude awakening. Geez, visual? You mean like SearchMe which went offline (with no one noticing)?