• http://sebastianx.blogspot.com/2007/04/in-need-of-web-robot-directives.html Sebastian

    The most important open issue is section targeting IMHO. Defining and standardizing robots.txt syntax to steer crawling on block level and even link attribute level would be a huge step forwards.

  • http://incredibill.blogspot.com/ IncrediBILL

    Both sitemaps and robots.txt are scraping vulnerabilities, I’ve seen it happen, so I’m a big fan of the authenticated crawl.

    However, your average webmaster won’t be able to stop abuse even with an authenticated crawl because most bots pretend to be visitors cloaked as Firefox and MSIE. Authenticated crawling would just be a simplification of keeping people from spoofing honest bots and the bad guys will just keep crawling.

    Solutions ARE coming, but the search engines can’t help you in this regard.