Where’s AP In Google News? Apparently In Limbo, As Contract Running Out

It’s been noticed that new Associated Press stories — hosted by Google itself — are no longer appearing in Google News. It’s true. Since Dec. 24, Google has no longer added new AP content, something the company confirmed with me today. I received this statement:

We have a licensing agreement with the Associated Press that permits us to host its content on Google properties such as Google News. Some of that content is still available today. At the moment we’re not adding new hosted content from the AP.

So why not? The statement doesn’t explain. But it’s reasonable to assume it’s related to the ongoing talks between Google and the Associated Press.

Google has an agreement to host AP articles on its own web site, plus to make use of AP material in other ways. That expires near the end of this month. Since the agreement only allows stories to be hosted for 30 days, it might be that Google’s covering the legal bases in case a new agreement isn’t reached. You don’t want a story going up on, say January 23, only to have to pull it down the next day.

Google News Now Hosting Wire Stories & Promises Better Variety In Results from us in 2007 has background about how Google began hosting stories from several wire services, including the AP, on its own site.

The AP, in particular, wanted its stories hosted. Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers from us last November explains more about this, how Google makes use of AP content under the current agreement and some of the issues that have come up in talks to strike a new deal.

It’s important to note that AP stories may still appear within Google. Many newspapers carry AP content, and those papers continue to be listed. So you can find AP stories hosted on newspaper sites. You just won’t find them hosted within Google itself.

On a somewhat related note, Rupert Murdoch’s News Corporation has threated to block major search engines, including Google, from crawling its news content. Some see the first shot in that threat being fired as UK-based The Times is now blocking the UK-based NewsNow search engine.

As it turns out, this seems largely unrelated to Murdoch’s complaints with Google. Instead, it focuses on NewsNow providing a commercial service that allows companies to monitor the news. The UK’s Newspaper Licensing Agency wants to charge companies that provide this type of service. While Murdoch’s Times hasn’t joined that push, which is under review by UK authorities, it has restricted NewsNow for the same reasons. PaidContent provides an excellent rundown on the situation in these articles:

Technically, NewsNow doesn’t have to obey the restrictions blocking it in the robots.txt file at The Times. It’s not a legally-binding protocol. But respected crawlers do obey it, which is one reason why there have been so few lawsuits over crawling.

That robots.txt file is farcical in one respect. At the top, it says this:

#Robots.txt File
#Version: 0.8
#Last updated: 04/01/2010
#Site contents Copyright Times Newspapers Ltd
#Please note our terms and conditions


#Spidering is not allowed by our terms and conditions
#Authorised spidering is subject to permission
#For authorisation please contact us -
 see http://www.nisyndication.com/about_us.html

A robots.txt file is designed to be read automatically by a machine. Anything line that begins with a # symbol is a comment that machines ignore (the two indented lines above without a # symbol are actually part of the line above, just with a line break in this story, so you can read them properly). There’s no way for Google or any search engine to automatically know this file that “Authorised spidering is subject to permission.”

The way they actually know this is if the robots.txt file uses an accepted command to block them — which it doesn’t, except for NewsNow and some smaller crawlers.

I highly doubt Google, Yahoo or Bing have actually asked The Times for permission to crawl them. But I have an email out to The Times and News Corporation to find out. Of course, if any human had tried to follow the link to seek authorization, they wouldn’t have gotten an error. That link doesn’t work.

Related Topics: Channel: Content | Features: Analysis | Google: Business Issues | Google: News | Top News


About The Author: is a Founding Editor of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also serves as Chief Content Officer for Third Door Media, which publishes Search Engine Land and produces the SMX: Search Marketing Expo conference series. He has a personal blog called Daggle (and keeps his disclosures page there). He can be found on Facebook, Google + and microblogs on Twitter as @dannysullivan.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide