Apple Webcrawler: More Potential Evidence Of Search Ambitions

Apple Insider reports on the discovery of a web-crawling bot originating from Apple’s servers. It was first “outed” by developer Jan Moesen.

This is what Moesen saw:

Moesen reports that the bot is only crawling HTML, “not the CSS, JavaScript or image files.” Then he asks whether this is an “official Apple project” or just “someone crawling the web from their workplace at Apple?”

I can’t answer Moesen’s question but I’m going to guess it’s an official Apple project. Interestingly, he says it has some sort of bug.

Apple has been working on “search” in various forms  for some time. Siri, though not a search engine, is a kind of replacement for search for certain types of queries and activities. Apple has been relying on Bing for websearch “backfill.”

In 2012 the company hired William Stasior from Amazon/A9. Before working at the Amazon search division, Stasior was Alta Vista’s “director of advanced development.” There he “led the engineering team responsible for developing AltaVista’s next generation search technologies.” 

Apple Maps is a local search engine. Apple Watch extends that local search functionality to your wrist.

In the Yosemite update to Mac OS the new Spotlight Search is front and center on the desktop. Spotlight searches your desktop but also provides web search suggestions from Bing. There are structured data sources that also show up in search results, such as Wikipedia, Maps and Fandango. 

Some of this replaces the need to go to Google, but only at the margins.

In this larger context my guess is that Apple is doing something purposeful with a webcrawler. I don’t think that Apple will ever take on Google directly by trying to be a general or all-purpose search engine, but web search and related content capabilities are an increasingly important part of the virtual assistant experience.

Accordingly I would argue that Apple needs more search chops and content if it is to further develop Spotlight Search and to keep Siri competitive with Google/Now and Cortana.

About The Author

Greg Sterling

Greg Sterling is a Contributing Editor at Search Engine Land. He writes a personal blog, Screenwerk, about connecting the dots between digital media and real-world consumer behavior. He is also VP of Strategy and Insights for the Local Search Association. Follow him on Twitter or find him at Google+.