Up Close With Google’s Search By Image: Hits, Misses & More

Earlier this week at its Inside Search event, Google announced a new way to search the web called Search By Image. Rather than enter a text query, Search By Image lets you begin your search with an image and Google will (try to) tell you what it is and show visually similar images.

Search By Image isn’t the first start-with-an-image search engine, and isn’t even Google’s first version of this kind of search tool. (TinEye has offered reverse image search since May 2008, and Google Goggles was Google’s first search-by-image tool back in 2009.) But Google is the big and new kid on the block, so we’ve taken an in-depth look at how Search By Image works — including when it’s at its best and worst.

How Google Search By Image Works

Search By Image is available now at images.google.com or via the “Images” tab in the left-side menu on Google.com. You should see a small camera icon on the far right side of the search bar.

There are several ways to access it:

• Drag and drop an image on the search bar
• Click the camera icon to upload an image from your computer
• Paste the URL of a photo on the web into the search bar
• Use the Chrome or Firefox extensions that add a search option to your computer’s contextual menu (right click)

After you supply a photo, Google goes to work. The company’s blog post explains that Google analyzes the image, creating a mathematical model based on shapes, lines, proportions, colors and other elements. It then matches the model against images already in Google’s index. Google then does page analysis to take a text-based guess at what the image is, which is part of the process of identifying the image and returning similar results.

Search By Image looks for similar content on the web, so unique or never-before-seen images won’t work well.

No Facial Recognition

One thing that Google emphasized Tuesday is that Search By Image does not use facial recognition. Although Google’s Picasa photo service does facial recognition and others like Exalead do, too, it’s become a hot-button issue on the privacy front. Facebook’s recent announcement that it would use facial recognition to identify friends in newly uploaded photos is the subject of government investigations, and Google has gone to great lengths to deny any plans of using the technology.

So, when you upload a mug shot, Google’s Search By Image results are interesting:

When I upload a photo of myself, Google is able to find web pages with the same image (1), even some that are cropped down the middle. But without using facial recognition, Google only shows similar images (2) that have a similar combination of shape, proportion and colors to my original. (Should I be bothered that a stuffed monkey shows up on row two as visually similar to me? Thanks for nothing, Google.)

By the way, the filename doesn’t appear to matter when you upload an image. I uploaded that mugshot of myself as yDSC_6801.jpg and as matt-mcgee.jpg, and both searches returned the exact same results.

But the browser you use might make a difference. The screenshot above is from Firefox — for some reason, the “visually similar images” returned were slightly different when I uploaded the same mugshot in Safari. Have a look:

See also our follow-up story,Disturbing Michelle Obama Image Makes A Case For Facial Recognition In Google’s New Image Search.

Search By Image: The Hits & Misses

If there’s no facial recognition, identifying people may not be the most recommended way to use Search By Image. So, how is this new feature supposed to be used? In the intro video, Google suggests that it’s for identifying “places, art and even mysterious creatures.”

So, let’s try a few sample images and see how well Google does, shall we?

Identifying Places

Death Valley, California, is one of my favorite places on Earth. (You have to visit it if you’ve never been there.) Zabriskie Point is one of the most well-known and photographed spots in Death Valley. I expected Search By Image to recognize one of my photos of it, and Google didn’t disappoint.

I thought that the Rock and Roll Hall of Fame and Museum in Cleveland might be a little harder to match, but Search By Image nailed it.

So, let’s make this more difficult. Another favorite place is the Pro Football Hall of Fame in Canton, Ohio. You’ve probably seen it from the outside — it has a half-dome above the entryway that almost looks like a church building. When I visited there last summer, I took a photo looking up at the inside of the half-dome. Search By Image was a complete miss on this one.

Notice the text above my uploaded photo? It’s asking me to help by providing a description. When I add “football hall of fame” to the search box and try again, Search By Image does better. It finds the official website and the (obligatory) Wikipedia page, and also locates a couple similar images (one of which is my photo uploaded to Flickr a year ago).

Still, that seems like “cheating” when you have to add some text to get the image matching correct, doesn’t it? Let’s try a couple other kinds of images.

Identifying Flowers

I have a thing for flower photos. Problem is, I can’t identify about 90% of what I shoot. Like this yellow flower that I’m sure everyone reading this can identify … but can Google?

Nope. I’m not the only one who doesn’t know what it is. (If you know, leave me a comment, please?)

Another miss: I was surprised to see that Search By Image couldn’t identify this photo of a sunflower.

Search By Image got the color right, and recognized the circular shape inside the photo. But I’m guessing the original photo was too macro to identify as a sunflower; something showing the entire flower might’ve worked better. (But I don’t have any such photos.)

Two Final Challenges

I grabbed two images from my collection that I thought would be real challenges for Search By Image. The first is a photo I took at Disney’s California Adventure — it’s Roz, the hysterical character from Monsters Inc. She’s the very definition of a “mysterious creature,” which is how Google suggests I use Search By Image! Sure, I know there’s no facial recognition, but in her case … might that actually help? Not really.

Apologies to all of the wrestlers and everyone else that was identified as “visually similar” to Roz. But that’s Google, not me. You can reach them at 1600 Amphitheatre Parkway, Mountain View, CA 94043.

My last image is a fairly nondescript photo of the Seattle skyline that I took from atop the Space Needle. Since the very distinctive and oft-photographed Space Needle wasn’t in the photo, I expected this might be tough to distinguish from any other city skyline. Boy, was I wrong.

That’s pretty impressive to me: Best guess for this image: seattle skyline? Yep. Exactly. Well played, Search By Image. I’m guessing it helps that a lot of people have taken similar images from atop the Space Needle.

What About TinEye?

TinEye has been offering similar image search capabilities for a couple years and boasts that it has an index of about two billion photos right now. One big difference is that TinEye doesn’t provide text or search results that tell you what you’re looking at; it only returns similar images.

But how well does it do that compared to Google?

I uploaded each of the same images into TinEye to see how well it did and here — minus what would be an overload of screenshots — are the results:

• Matt McGee mugshot – TinEye correctly identified me by name and displayed several exact matching images
• Zabriskie Point – correctly returned a set of matching images
• Rock and Roll Hall of Fame and Museum – TinEye couldn’t find any similar/matching images
• Pro Football Hall of Fame interior – no similar/matching images
• mystery yellow flower – no similar/matching images
• sunflower macro – no similar/matching images
• Roz – no similar/matching images
• Seattle skyline – correctly identified and displayed similar/matching images

Final Thoughts

Both Google’s Search By Image and TinEye performed very similarly with the images that I uploaded from my collection: They did very well for obvious landmarks and things that are commonly photographed and found online, and not very well for other types of images. The Rock Hall building is the only image that Google identified and TinEye didn’t.

But my sample was extremely limited and designed just to get a general sense of how Google’s new feature works, and how it compares to one that’s been available for years now. To get a more accurate sense of the two, you’d need to do probably dozens of more uploads and more detailed testing on whether things like image size matter (my colleague Gary Price thinks it does, based on some of his own tests with Search By Image).

For me, the final verdict is this: If you need to know what an image is and plan to use Google’s Search By Image or TinEye, you better hope there are already a lot of similar images online and, in Google’s case, a lot of web pages (preferably one from Wikipedia) that describe the image.

Postscript: See our follow-up post, Disturbing Michelle Obama Image Makes A Case For Facial Recognition In Google’s New Image Search.