Apple Visual Intelligence – search using your iPhone camera

Apple uses Google, OpenAI and other search providers for the search results.

Chat with SearchBot

Apple demonstrated a new search experience with Apple Intelligence named Apple Visual Intelligence. It looks and feels like Google Lens but it uses the native iPhone camera and is built into Apple Intelligence directly.

Plus, it seems to use third-party search providers, like Google, OpenAI’s ChatGPT and Yelp for its search results – depending on the type of query.

What it looks like. Here are some screenshots I grabbed from the Apple event from yesterday, if you want to watch it, it starts at about the 57-minute mark in this video:

Looking to buy a bike you saw on your walk; it says “Searching with Google…” after you snap a photo of it:

Apple Visual Intelligence Camera Google Shopping 1725909323 Scaled

Although, the example provided of the search results look somewhat “doctored”:

Apple Visual Intelligence Camera Google Shopping 2 1725909323 Scaled

Here is an example of a local search result when someone wants more details on a restaurant they came across while walking. This seems to pull up the local search results in Apple Maps, which I believe is powered by Yelp and OpenTable.

Apple Visual Intelligence Camera Local 1725909323 1 Scaled

Here is a close up showing OpenTable options in Apple Maps:

Apple Visual Intelligence Camera Local2 1725909323 Scaled

Then here is an example of taking a photo of a homework assignment, where it uses OpenAI’s ChatGPT for help:

Apple Visual Intelligence Camera Chatgpt 1725909323 Scaled

Why we care. Apple seems to be using AI as a tool rather than a foundation for its devices, where it integrates with Google, OpenAI and other search providers. There is obviously underlining AI and machine learning that is taking place on the Apple device, but the results seem to be coming from third-parties.

An early beta review from the Washington Post suggests it has a long way to go. Specifically it has issues with with hallucinations, marking spam emails as priority, and other problems.


About the author

Barry Schwartz
Staff
Barry Schwartz is a technologist and a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics.

In 2019, Barry was awarded the Outstanding Community Services Award from Search Engine Land, in 2018 he was awarded the US Search Awards the "US Search Personality Of The Year," you can learn more over here and in 2023 he was listed as a top 50 most influential PPCer by Marketing O'Clock.

Barry can be followed on X here and you can learn more about Barry Schwartz over here or on his personal site.

Get the newsletter search marketers rely on.