Google Releases Details On Controlling GoogleBot & Google’s Crawl
The Google Webmaster Central blog announced Google has published a new set of documents in the Google Code section on how to control Google from crawling and indexing your site. You can read the set of documents over here. The technical documents are broken down into five section: Getting Started Robots.txt specification Robots meta tag […]
Barry Schwartz on November 26, 2010 at 8:47 am | Reading time: 1 minute
The Google Webmaster Central blog announced Google has published a new set of documents in the Google Code section on how to control Google from crawling and indexing your site. You can read the set of documents over here.
The technical documents are broken down into five section:
- Getting Started
- Robots.txt specification
- Robots meta tag and X-Robots-Tag specification
- Google’s crawlers
- References
I personally printed them out as my weekend reading.
Related stories