« Hitwise: Google Hits New High; Microsoft & Yahoo New Lows | Main | SearchCap: The Day In Search, April 14, 2008 »
Apr. 14, 2008 at 4:00pm Eastern by Barry Schwartz
Yahoo Updates Crawler: Introduces Yahoo Slurp 3.0
Yahoo has announced the launch of their new crawler, Yahoo! Slurp 3.0. 3.0 is slowly being pushed out over the course of the "next several weeks," including "some infrastructure updates," Yahoo said. I suspect the recent changes we reported with the Yahoo Search Tax Day update is related to the "infrastructure updates."
The new spider will contain the same user-agent and all robots.txt directives, but will be named "Slurp 3.0" as opposed to "Slurp 2.0." Yahoo notifies us to expect two changes:
(1) The crawlers will reside on a new set of IP addresses, but will still reverse DNS lookup to the crawl.yahoo.net domain. If you are using IP-based rules, make sure to keep an eye out for the new Slurp crawler and add those sets of IP addresses to your database.
(2) The new crawlers will return a new user agent, specifically Yahoo! Slurp/3.0. If you set directives for "Slurp" or "Yahoo! Slurp," those will remain working, but if you specified "Slurp/2.0," then you may have an issue. Yahoo recommends using User-agent: Slurp in your robots.txt file.
Remember, you can reverse DNS lookup the spiders to confirm that this new crawler is from Yahoo. If you have any problems, Yahoo asks you submit them at the SiteExplorer forum.
This update confirmed two past posts:
(1) Yahoo Slurp being extremely hyperactive one weekend.
(2) A Yahoo Search update that we reported over the weekend.
|
Like The Story? Vote For It On Yahoo Buzz!
Send me the monthly search newsletter too! (Learn more about our newsletters and feeds) |
Subscribe To Our Search Feed!         
|
| Share & Bookmark This Story! |
By Barry Schwartz
Permalink
Jump To Comments
See Related Stories In: Yahoo: SEO, Yahoo: Search
