Yahoo Updates Crawler: Introduces Yahoo Slurp 3.0

Yahoo has announced the launch of their new crawler, Yahoo! Slurp 3.0. 3.0 is slowly being pushed out over the course of the “next several weeks,” including “some infrastructure updates,” Yahoo said. I suspect the recent changes we reported with the Yahoo Search Tax Day update is related to the “infrastructure updates.”

The new spider will contain the same user-agent and all robots.txt directives, but will be named “Slurp 3.0″ as opposed to “Slurp 2.0.” Yahoo notifies us to expect two changes:

(1) The crawlers will reside on a new set of IP addresses, but will still reverse DNS lookup to the domain. If you are using IP-based rules, make sure to keep an eye out for the new Slurp crawler and add those sets of IP addresses to your database.

(2) The new crawlers will return a new user agent, specifically Yahoo! Slurp/3.0. If you set directives for “Slurp” or “Yahoo! Slurp,” those will remain working, but if you specified “Slurp/2.0,” then you may have an issue. Yahoo recommends using User-agent: Slurp in your robots.txt file.

Remember, you can reverse DNS lookup the spiders to confirm that this new crawler is from Yahoo. If you have any problems, Yahoo asks you submit them at the SiteExplorer forum.

This update confirmed two past posts:

(1) Yahoo Slurp being extremely hyperactive one weekend. (2) A Yahoo Search update that we reported over the weekend.

Related Topics: Channel: SEO | Yahoo: Search | Yahoo: SEO


About The Author: is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here. For more background information on Barry, see his full bio over here.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide