Yahoo Updates Crawler: Introduces Yahoo Slurp 3.0

Apr 14, 2008 at 4:00pm ET by Barry Schwartz

Yahoo has announced the launch of their new crawler, Yahoo! Slurp 3.0. 3.0 is slowly being pushed out over the course of the “next several weeks,” including “some infrastructure updates,” Yahoo said. I suspect the recent changes we reported with the Yahoo Search Tax Day update is related to the “infrastructure updates.”

The new spider will contain the same user-agent and all robots.txt directives, but will be named “Slurp 3.0″ as opposed to “Slurp 2.0.” Yahoo notifies us to expect two changes:

(1) The crawlers will reside on a new set of IP addresses, but will still reverse DNS lookup to the crawl.yahoo.net domain. If you are using IP-based rules, make sure to keep an eye out for the new Slurp crawler and add those sets of IP addresses to your database.

(2) The new crawlers will return a new user agent, specifically Yahoo! Slurp/3.0. If you set directives for “Slurp” or “Yahoo! Slurp,” those will remain working, but if you specified “Slurp/2.0,” then you may have an issue. Yahoo recommends using User-agent: Slurp in your robots.txt file.

Remember, you can reverse DNS lookup the spiders to confirm that this new crawler is from Yahoo. If you have any problems, Yahoo asks you submit them at the SiteExplorer forum.

This update confirmed two past posts:

(1) Yahoo Slurp being extremely hyperactive one weekend. (2) A Yahoo Search update that we reported over the weekend.


Barry Schwartz is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here.

See more articles by Barry Schwartz >

Share, Bookmark & Discuss This Article
More:

Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter

See more stories like this in the Members Library! Check out the Yahoo: SEO, Yahoo: Search sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!

Comments are closed.

RECENT COMMENTS

  • Susannah said " I can't wait to try some of these tips this week. What a resource! It's like having a coffee with 21"
  • dian said " I haven't tried that yet but if it is the way Mazter is saying I think it won't going to do any good"
  • dian said " It really helps me a lot. The methods step by step explained the important factors easily and all th"

See All »

FREE DAILY SEARCH NEWS RECAP!

Stay on top of all the search news with our daily summary, the SearchCap newsletter. View a sample ›

STAY CURRENT THROUGHOUT THE DAY

RSS Feeds

The Search Engine Land feed keeps you informed as news happens. SEE ALL FEEDS »

Upcoming Search Engine Land Conferences

Advertise With Us »

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.


SMX Web Site » | SMX Difference » | SMX News »


Join us at an upcoming SMX event:

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:


See more webcast topics »

TRACK US SOCIALLY
Upcoming Search Engine Land Conferences

Get Your Search Engine Land
Premium Membership!

Become a premium member today and receive:

  • Express commenting privileges & photo.
  • Exclusive videos & newsletters.
  • Discounts to our SMX conferences.
  • Access to "How To" & Other Archives.

Learn More

Upcoming Search Engine Land Conferences
En Espanol|Contact|Members Info|RSS Feeds|Email Subscriptions|Privacy|Advertise
Add to GoogleAdd to My Yahoo!Add to BloglinesAdd to NetvibesAdd to Windows Live