Comments on: Up Close & Personal With Robots.txt http://searchengineland.com/up-close-personal-with-robotstxt-10978 Search Engine Land: Must Read News About Search Marketing & Search Engines Mon, 23 Nov 2009 01:49:26 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: IncrediBILL http://searchengineland.com/up-close-personal-with-robotstxt-10978/comment-page-1#comment-1977 IncrediBILL Tue, 17 Apr 2007 01:30:23 +0000 http://searchengineland.com/beta/up-close-personal-with-robotstxt-10978.php#comment-1977 Both sitemaps and robots.txt are scraping vulnerabilities, I've seen it happen, so I'm a big fan of the authenticated crawl. However, your average webmaster won't be able to stop abuse even with an authenticated crawl because most bots pretend to be visitors cloaked as Firefox and MSIE. Authenticated crawling would just be a simplification of keeping people from spoofing honest bots and the bad guys will just keep crawling. Solutions ARE coming, but the search engines can't help you in this regard. Both sitemaps and robots.txt are scraping vulnerabilities, I’ve seen it happen, so I’m a big fan of the authenticated crawl.

However, your average webmaster won’t be able to stop abuse even with an authenticated crawl because most bots pretend to be visitors cloaked as Firefox and MSIE. Authenticated crawling would just be a simplification of keeping people from spoofing honest bots and the bad guys will just keep crawling.

Solutions ARE coming, but the search engines can’t help you in this regard.

]]> By: Sebastian http://searchengineland.com/up-close-personal-with-robotstxt-10978/comment-page-1#comment-1976 Sebastian Mon, 16 Apr 2007 17:56:30 +0000 http://searchengineland.com/beta/up-close-personal-with-robotstxt-10978.php#comment-1976 The most important open issue is section targeting IMHO. Defining and standardizing robots.txt syntax to steer crawling on block level and even link attribute level would be a huge step forwards. The most important open issue is section targeting IMHO. Defining and standardizing robots.txt syntax to steer crawling on block level and even link attribute level would be a huge step forwards.

]]>