Search Engine Land
  • SEO
    • > All SEO
    • > What Is SEO?
    • > SEO Periodic Table
    • > Google: SEO
    • > Bing SEO
    • > Google Algorithm Updates
  • PPC
    • > All PPC
    • > What is PPC?
    • > Google Ads
    • > Microsoft Ads
    • > The Periodic Tables of PPC
  • Focuses
    • > Local
    • > Commerce
    • > Shopify SEO Guide
    • > Content
    • > Email Marketing Periodic Table
    • > Social Media Marketing
    • > Analytics
    • > Search Engine Land Awards
    • > All Focuses
  • SMX
  • Webinars
  • Intelligence Reports
  • White Papers
  • About
    • > About Search Engine Land
    • > Newsletter
    • > Third Door Media
    • > Advertise

Processing...Please wait.

Search Engine Land » Channel » Content » Where’s AP In Google News? Apparently In Limbo, As Contract Running Out

Where’s AP In Google News? Apparently In Limbo, As Contract Running Out

It’s been noticed that new Associated Press stories — hosted by Google itself — are no longer appearing in Google News. It’s true. Since Dec. 24, Google has no longer added new AP content, something the company confirmed with me today. I received this statement: We have a licensing agreement with the Associated Press that […]

Danny Sullivan on January 8, 2010 at 7:21 pm

It’s been noticed that new Associated Press stories — hosted by Google itself — are no longer appearing in Google News. It’s true. Since Dec. 24, Google has no longer added new AP content, something the company confirmed with me today. I received this statement:

We have a licensing agreement with the Associated Press that permits us to host its content on Google properties such as Google News. Some of that content is still available today. At the moment we’re not adding new hosted content from the AP.

So why not? The statement doesn’t explain. But it’s reasonable to assume it’s related to the ongoing talks between Google and the Associated Press.

Google has an agreement to host AP articles on its own web site, plus to make use of AP material in other ways. That expires near the end of this month. Since the agreement only allows stories to be hosted for 30 days, it might be that Google’s covering the legal bases in case a new agreement isn’t reached. You don’t want a story going up on, say January 23, only to have to pull it down the next day.

Google News Now Hosting Wire Stories & Promises Better Variety In Results from us in 2007 has background about how Google began hosting stories from several wire services, including the AP, on its own site.

The AP, in particular, wanted its stories hosted. Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers from us last November explains more about this, how Google makes use of AP content under the current agreement and some of the issues that have come up in talks to strike a new deal.

It’s important to note that AP stories may still appear within Google. Many newspapers carry AP content, and those papers continue to be listed. So you can find AP stories hosted on newspaper sites. You just won’t find them hosted within Google itself.

On a somewhat related note, Rupert Murdoch’s News Corporation has threated to block major search engines, including Google, from crawling its news content. Some see the first shot in that threat being fired as UK-based The Times is now blocking the UK-based NewsNow search engine.

As it turns out, this seems largely unrelated to Murdoch’s complaints with Google. Instead, it focuses on NewsNow providing a commercial service that allows companies to monitor the news. The UK’s Newspaper Licensing Agency wants to charge companies that provide this type of service. While Murdoch’s Times hasn’t joined that push, which is under review by UK authorities, it has restricted NewsNow for the same reasons. PaidContent provides an excellent rundown on the situation in these articles:

  • Murdoch Paper Blocks UK Aggregator Before Paywall Goes Up
  • UK Newspapers Suspend ‘Link Tax’ Bills To End Users

Technically, NewsNow doesn’t have to obey the restrictions blocking it in the robots.txt file at The Times. It’s not a legally-binding protocol. But respected crawlers do obey it, which is one reason why there have been so few lawsuits over crawling.

That robots.txt file is farcical in one respect. At the top, it says this:

#Robots.txt File
#Version: 0.8
#Last updated: 04/01/2010
#Site contents Copyright Times Newspapers Ltd
#Please note our terms and conditions
 http://www.timesonline.co.uk/section/0,,497,00.html
#Spidering is not allowed by our terms and conditions
#Authorised spidering is subject to permission
#For authorisation please contact us -
 see http://www.nisyndication.com/about_us.html

A robots.txt file is designed to be read automatically by a machine. Anything line that begins with a # symbol is a comment that machines ignore (the two indented lines above without a # symbol are actually part of the line above, just with a line break in this story, so you can read them properly). There’s no way for Google or any search engine to automatically know this file that “Authorised spidering is subject to permission.”

The way they actually know this is if the robots.txt file uses an accepted command to block them — which it doesn’t, except for NewsNow and some smaller crawlers.

I highly doubt Google, Yahoo or Bing have actually asked The Times for permission to crawl them. But I have an email out to The Times and News Corporation to find out. Of course, if any human had tried to follow the link to seek authorization, they wouldn’t have gotten an error. That link doesn’t work.


Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.


New on Search Engine Land

    GA4 isn’t all it’s cracked up to be. What would it look like to switch?

    What to look for in a technical SEO audit

    What are your secrets to overcoming marketing challenges? Take our survey

    10 reasons to join us at SMX Advanced online this June

    How to ensure influencers help your SEO campaigns

About The Author

Danny Sullivan
Danny Sullivan was a journalist and analyst who covered the digital and search marketing space from 1996 through 2017. He was also a cofounder of Third Door Media, which publishes Search Engine Land and MarTech, and produces the SMX: Search Marketing Expo and MarTech events. He retired from journalism and Third Door Media in June 2017. You can learn more about him on his personal site & blog He can also be found on Facebook and Twitter.

Related Topics

ContentGoogleGoogle NewsSEO

Get the daily newsletter search marketers rely on.

Processing...Please wait.

See terms.

ATTEND OUR EVENTS

Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.

March 8-9, 2022: Master Classes (virtual)

June 14-15, 2022: SMX Advanced (virtual)

November 15-16, 2022: SMX Next (virtual)

Learn More About Our SMX Events

Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.

Start Discovering Now: Spring (virtual)

September 28-29, 2022: Fall (virtual)

Learn More About Our MarTech Events

Webinars

Data-Driven Answers to Achieve Omnichannel Success

Is Your Marketing Stack Ready for Omnichannel CX?

Outrank in Organic Search with These 5 Core Tactics

See More Webinars

Intelligence Reports

Enterprise SEO Platforms: A Marketer’s Guide

Enterprise Identity Resolution Platforms

Email Marketing Platforms: A Marketer’s Guide

Enterprise Sales Enablement Platforms: A Marketer’s Guide

Enterprise Digital Experience Platforms: A Marketer’s Guide

Enterprise Call Analytics Platforms: A Marketer’s Guide

See More Intelligence Reports

White Papers

Realising the power of virtual events for demand generation

The Progressive Marketer’s Ultimate Events Strategy 2022 Worksheet

CMO Guide: How to Plan Smart and Pivot Fast

The Retail Renaissance Report, USA Edition: 4 Keys to Predicting Online & In-Store Demand Across Global Markets

Thinking Bigger About Marketing Budgets

See More Whitepapers

Receive daily search news and analysis.

Processing...Please wait.

Topics

  • SEO
  • PPC

Our Events

  • Search Marketing Expo - SMX
  • MarTech

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS
  • Youtube

© 2022 Third Door Media, Inc. All rights reserved.