Sign up for weekly recaps of the ever-changing search marketing landscape.
Live Search Testing New Crawler; MSNBot/2.0b
The Live Search Blog announced they are letting a new robot loose. The new search engine crawler is named msnbot/2.0b and will be added to the army of current MSN spiders, currently named msnbot/1.1.
The new spider is currently being tested but will ultimately replace the old spider. The new spider will respect the current robots.txt protocol set up for MSNBot, so no need to set up anything new in your robots.txt file. In addition, Microsoft promised to crawl slowly in their msnbot/2.0b tests.
MSNBot/1.1 is not that old. It was added back in February of this year and introduced HTTP compression and conditional gets.