Yahoo Search Weather Update & Support For X-Robots Tag

The Yahoo Blog issued a weather report for changes to rankings in Yahoo Search, along with news that they are now supporting the X-Robots-Tag directive — a way to control indexing of content that cannot accept meta robots tags. Google also supports X-Robots, which gives webmasters the ability to define robots.txt like rules within http […]

Chat with SearchBot

The Yahoo Blog issued a weather report for changes to rankings in Yahoo Search, along with news that they are now supporting the X-Robots-Tag directive — a way to control indexing of content that cannot accept meta robots tags.


Google also supports X-Robots, which gives webmasters the ability to define robots.txt like rules within http headers, as opposed to just the META data within HTML pages.

Yahoo provided a few examples of how it can work:

  • X-Robots-Tag: NOINDEX — If you don’t want to show the URL in the Yahoo! Search results. Note: We’ll still need to crawl the page to see and apply the tag, so if you don’t wish to have the page crawled, use robots disallow on robots.txt.
  • X-Robots-Tag: NOARCHIVE — If you don’t want to display the cache link in the search results page.
  • X-Robots-Tag: NOSNIPPET — If you don’t want to display a summary in the search results page.
  • X-Robots-Tag: NOFOLLOW — If you don’t want Yahoo! to crawl links in the page.

About the author

Barry Schwartz
Staff
Barry Schwartz is a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Get the must-read newsletter for search marketers.