Google Adds Googlebot-News User Agent To Allow Blocking Google News

Dec 2, 2009 at 11:05am ET by Barry Schwartz

Google announced they have added a new supported user agent that can be used to block Google from indexing stories specifically on Google News. You can now specific the user agent Googlebot-News in your robots.txt file to exclude documents from being crawled and displayed in Google News.

Google has several user agents for controlling how Google has access to your content. They have Googlebot, Googlebot-Image and so on. Each can be used to block specific content from showing up in specific Google search properties.

In an effort to please some of the news providers, Google announced yesterday an amendment to the first click free program, allowing publishers to restrict it to the first five free accesses per user each day. This is Google’s second move to show publishers how they can manage their content in Google News.

The Google Webmaster Blog post has a lot more technical details on how to use this new user agent in your robots.txt file.

Also see Josh Cohen Of Google News On Paywalls, Partnerships & Working With Publishers from last month, which goes into more detail about issues some publishers had about wanting to block just Google News automatically rather than all of Google. Google said it would consider this type of change in that interview.

Note that it was always possible to block Google News by making a manual request to Google. In fact, inclusion into Google News in the first place is done through a manual process, rather than automated.


Barry Schwartz is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here.

See more articles by Barry Schwartz >

Share, Bookmark & Discuss This Article
More:

Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter

See more stories like this in the Members Library! Check out the Google: News, Google: SEO, Google: Webmaster Central, SEO: Blocking Spiders, Top News sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!

Comments are closed.

RECENT COMMENTS

  • Carrie Hill said " Hi Fran, Did you click on the "comparison" view in the top right corner of the graphic - look at the"
  • stefanw said " I love it when great minds come together. And then I'm there to bring it all down. But seriously I c"
  • davebarnes said " Or you can use AdBlock Plus with this filter: *google-analytics.com*"

See All »

FREE DAILY SEARCH NEWS RECAP!

SearchCap is a once-per-day newsletter update:

STAY CURRENT THROUGHOUT THE DAY

Our feed & social options update you as news happens.


Advertise With Us »

Search Marketing Expo

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.


SMX Web Site » | SMX Difference » | SMX News »


Join us at an upcoming SMX event:

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:


See more webcast topics »

FOLLOW US SOCIALLY
Upcoming Search Engine Land Conferences

Get Your Search Engine Land
Premium Membership!

Become a premium member today and receive:

  • Express commenting privileges & photo.
  • Exclusive videos & newsletters.
  • Discounts to our SMX conferences.
  • Access to "How To" & Other Archives.

Learn More

Upcoming Search Engine Land Conferences
Contact|Members Info|RSS Feeds|Email Subscriptions|Privacy|Advertise
Add to GoogleAdd to My Yahoo!Add to BloglinesAdd to NetvibesAdd to Windows Live