Everything you need to know about SEO, delivered every Thursday.
Google Releases Details On Controlling GoogleBot & Google’s Crawl
The Google Webmaster Central blog announced Google has published a new set of documents in the Google Code section on how to control Google from crawling and indexing your site. You can read the set of documents over here.
The technical documents are broken down into five section:
- Getting Started
- Robots.txt specification
- Robots meta tag and X-Robots-Tag specification
- Google’s crawlers
I personally printed them out as my weekend reading.