The Yahoo Blog issued a weather report for changes to rankings in Yahoo Search, along with news that they are now supporting the X-Robots-Tag directive — a way to control indexing of content that cannot accept meta robots tags.
Google also supports X-Robots, which gives webmasters the ability to define robots.txt like rules within http headers, as opposed to just the META data within HTML pages.
Yahoo provided a few examples of how it can work:
- X-Robots-Tag: NOINDEX — If you don’t want to show the URL in the Yahoo! Search results. Note: We’ll still need to crawl the page to see and apply the tag, so if you don’t wish to have the page crawled, use robots disallow on robots.txt.
- X-Robots-Tag: NOARCHIVE — If you don’t want to display the cache link in the search results page.
- X-Robots-Tag: NOSNIPPET — If you don’t want to display a summary in the search results page.
- X-Robots-Tag: NOFOLLOW — If you don’t want Yahoo! to crawl links in the page.