Apple Confirms Their Web Crawler: Applebot

After much speculation around an Apple Web Crawler, Apple has finally posted a help document confirming the existence of AppleBot, their web crawler.

Apple said, Applebot is the web crawler for Apple. AppleBot is “used by products including Siri and Spotlight Suggestions,” the company said.

The user-agent will typically follow the following string but will contain “Applebot” in it always:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

Apple says it will respect the customary robots.txt rules and robots meta tags. AppleBot currently originates in the 17.0.0.0 net block. If you do not mention AppleBot in your robots.txt directive, Apple will follow what you mention for Googlebot. So if you want to block AppleBot and GoogleBot, you can just block GoogleBot, but I’d recommend you block each individually.

If you notice unusual AppleBot activity, you can reach Apple about it at Apple-NOC “at” apple.com.

It is unclear if Apple plans on building and competing with Google on search but this is one step closer to that.

Postscript: Apple Insider reported later on today that Apple is “rapidly-expanding internal search group” to build their own version of a web search engine via Spotlight. So it does appear Apple is venturing into Google’s search space – at least based on these early reports.

About The Author

Barry Schwartz

Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here.