Apple Confirms Their Web Crawler: Applebot

After much speculation around an Apple Web Crawler, Apple has finally posted a help document confirming the existence of AppleBot, their web crawler.

Apple said, Applebot is the web crawler for Apple. AppleBot is “used by products including Siri and Spotlight Suggestions,” the company said.

The user-agent will typically follow the following string but will contain “Applebot” in it always:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

Apple says it will respect the customary robots.txt rules and robots meta tags. AppleBot currently originates in the 17.0.0.0 net block. If you do not mention AppleBot in your robots.txt directive, Apple will follow what you mention for Googlebot. So if you want to block AppleBot and GoogleBot, you can just block GoogleBot, but I’d recommend you block each individually.

If you notice unusual AppleBot activity, you can reach Apple about it at Apple-NOC “at” apple.com.

It is unclear if Apple plans on building and competing with Google on search but this is one step closer to that.

Postscript: Apple Insider reported later on today that Apple is “rapidly-expanding internal search group” to build their own version of a web search engine via Spotlight. So it does appear Apple is venturing into Google’s search space – at least based on these early reports.