Google Adds Googlebot-News User Agent To Allow Blocking Google News

Google announced they have added a new supported user agent that can be used to block Google from indexing stories specifically on Google News. You can now specific the user agent Googlebot-News in your robots.txt file to exclude documents from being crawled and displayed in Google News.

Google has several user agents for controlling how Google has access to your content. They have Googlebot, Googlebot-Image and so on. Each can be used to block specific content from showing up in specific Google search properties.

In an effort to please some of the news providers, Google announced yesterday an amendment to the first click free program, allowing publishers to restrict it to the first five free accesses per user each day. This is Google’s second move to show publishers how they can manage their content in Google News.

The Google Webmaster Blog post has a lot more technical details on how to use this new user agent in your robots.txt file.

Also see Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers from last month, which goes into more detail about issues some publishers had about wanting to block just Google News automatically rather than all of Google. Google said it would consider this type of change in that interview.

Note that it was always possible to block Google News by making a manual request to Google. In fact, inclusion into Google News in the first place is done through a manual process, rather than automated.

Related Topics: Channel: SEO | Google: News | Google: SEO | Google: Webmaster Central | SEO: Blocking Spiders | Top News


About The Author: is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here. For more background information on Barry, see his full bio over here.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide