Robots.txt Celebrates 20 Years Of Blocking Search Engines

Today is the 20th anniversary of the robots.txt directive being available for webmasters to block search engines from crawling their pages. The robots.txt was created by Martijn Koster in 1994 while he was working at Nexor after having issues with crawlers hitting his sites too hard. All major search engines back then, including WebCrawler, Lycos […]

Chat with SearchBot

googlebot-at-the-beach-1403523511Today is the 20th anniversary of the robots.txt directive being available for webmasters to block search engines from crawling their pages.

The robots.txt was created by Martijn Koster in 1994 while he was working at Nexor after having issues with crawlers hitting his sites too hard. All major search engines back then, including WebCrawler, Lycos and AltaVista, quickly adopted it; and even 20 years later, all major search engines continue to support it and obey it.

Brian Ussery posted on his blog about the 20-year anniversary and documented the most common robots.txt mistakes he has seen over his SEO tenure. It is well worth scanning through, because the robots.txt, if implemented wrong, can be severely detrimental to your rankings and search marketing success.

For more details on the robots.txt see robotstxt.org, Wikipedia and Google support. Also, make sure to check out our blocking spiders section.


About the author

Barry Schwartz
Staff
Barry Schwartz is a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Get the must-read newsletter for search marketers.