Up Close With Google’s Search By Image: Hits, Misses & More

search-by-image-logoEarlier this week at its Inside Search event, Google announced a new way to search the web called Search By Image. Rather than enter a text query, Search By Image lets you begin your search with an image and Google will (try to) tell you what it is and show visually similar images.

Search By Image isn’t the first start-with-an-image search engine, and isn’t even Google’s first version of this kind of search tool. (TinEye has offered reverse image search since May 2008, and Google Goggles was Google’s first search-by-image tool back in 2009.) But Google is the big and new kid on the block, so we’ve taken an in-depth look at how Search By Image works — including when it’s at its best and worst.

How Google Search By Image Works

Search By Image is available now at images.google.com or via the “Images” tab in the left-side menu on Google.com. You should see a small camera icon on the far right side of the search bar.

There are several ways to access it:

  • Drag and drop an image on the search bar
  • Click the camera icon to upload an image from your computer
  • Paste the URL of a photo on the web into the search bar
  • Use the Chrome or Firefox extensions that add a search option to your computer’s contextual menu (right click)

After you supply a photo, Google goes to work. The company’s blog post explains that Google analyzes the image, creating a mathematical model based on shapes, lines, proportions, colors and other elements. It then matches the model against images already in Google’s index. Google then does page analysis to take a text-based guess at what the image is, which is part of the process of identifying the image and returning similar results.

Search By Image looks for similar content on the web, so unique or never-before-seen images won’t work well.

No Facial Recognition

One thing that Google emphasized Tuesday is that Search By Image does not use facial recognition. Although Google’s Picasa photo service does facial recognition and others like Exalead do, too, it’s become a hot-button issue on the privacy front. Facebook’s recent announcement that it would use facial recognition to identify friends in newly uploaded photos is the subject of government investigations, and Google has gone to great lengths to deny any plans of using the technology.

So, when you upload a mug shot, Google’s Search By Image results are interesting:

search-by-image-1

When I upload a photo of myself, Google is able to find web pages with the same image (1), even some that are cropped down the middle. But without using facial recognition, Google only shows similar images (2) that have a similar combination of shape, proportion and colors to my original. (Should I be bothered that a stuffed monkey shows up on row two as visually similar to me? Thanks for nothing, Google.)

By the way, the filename doesn’t appear to matter when you upload an image. I uploaded that mugshot of myself as yDSC_6801.jpg and as matt-mcgee.jpg, and both searches returned the exact same results.

But the browser you use might make a difference. The screenshot above is from Firefox — for some reason, the “visually similar images” returned were slightly different when I uploaded the same mugshot in Safari. Have a look:

search-by-image-2

See also our follow-up story,Disturbing Michelle Obama Image Makes A Case For Facial Recognition In Google’s New Image Search.

Search By Image: The Hits & Misses

If there’s no facial recognition, identifying people may not be the most recommended way to use Search By Image. So, how is this new feature supposed to be used? In the intro video, Google suggests that it’s for identifying “places, art and even mysterious creatures.”

So, let’s try a few sample images and see how well Google does, shall we?

Identifying Places

Death Valley, California, is one of my favorite places on Earth. (You have to visit it if you’ve never been there.) Zabriskie Point is one of the most well-known and photographed spots in Death Valley. I expected Search By Image to recognize one of my photos of it, and Google didn’t disappoint.

search-by-image-3

I thought that the Rock and Roll Hall of Fame and Museum in Cleveland might be a little harder to match, but Search By Image nailed it.

search-by-image-4

So, let’s make this more difficult. Another favorite place is the Pro Football Hall of Fame in Canton, Ohio. You’ve probably seen it from the outside — it has a half-dome above the entryway that almost looks like a church building. When I visited there last summer, I took a photo looking up at the inside of the half-dome. Search By Image was a complete miss on this one.

search-by-image-5

Notice the text above my uploaded photo? It’s asking me to help by providing a description. When I add “football hall of fame” to the search box and try again, Search By Image does better. It finds the official website and the (obligatory) Wikipedia page, and also locates a couple similar images (one of which is my photo uploaded to Flickr a year ago).

search-by-image-6

Still, that seems like “cheating” when you have to add some text to get the image matching correct, doesn’t it? Let’s try a couple other kinds of images.

Identifying Flowers

I have a thing for flower photos. Problem is, I can’t identify about 90% of what I shoot. Like this yellow flower that I’m sure everyone reading this can identify … but can Google?

search-by-image-7

Nope. I’m not the only one who doesn’t know what it is. (If you know, leave me a comment, please?)

Another miss: I was surprised to see that Search By Image couldn’t identify this photo of a sunflower.

search-by-image-8

Search By Image got the color right, and recognized the circular shape inside the photo. But I’m guessing the original photo was too macro to identify as a sunflower; something showing the entire flower might’ve worked better. (But I don’t have any such photos.)

Two Final Challenges

I grabbed two images from my collection that I thought would be real challenges for Search By Image. The first is a photo I took at Disney’s California Adventure — it’s Roz, the hysterical character from Monsters Inc. She’s the very definition of a “mysterious creature,” which is how Google suggests I use Search By Image! Sure, I know there’s no facial recognition, but in her case … might that actually help? Not really.

search-by-image-9

Apologies to all of the wrestlers and everyone else that was identified as “visually similar” to Roz. But that’s Google, not me. You can reach them at 1600 Amphitheatre Parkway, Mountain View, CA 94043.

My last image is a fairly nondescript photo of the Seattle skyline that I took from atop the Space Needle. Since the very distinctive and oft-photographed Space Needle wasn’t in the photo, I expected this might be tough to distinguish from any other city skyline. Boy, was I wrong.

search-by-image-10

That’s pretty impressive to me: Best guess for this image: seattle skyline? Yep. Exactly. Well played, Search By Image. I’m guessing it helps that a lot of people have taken similar images from atop the Space Needle.

What About TinEye?

TinEye has been offering similar image search capabilities for a couple years and boasts that it has an index of about two billion photos right now. One big difference is that TinEye doesn’t provide text or search results that tell you what you’re looking at; it only returns similar images.

But how well does it do that compared to Google?

I uploaded each of the same images into TinEye to see how well it did and here — minus what would be an overload of screenshots — are the results:

  • Matt McGee mugshot – TinEye correctly identified me by name and displayed several exact matching images
  • Zabriskie Point – correctly returned a set of matching images
  • Rock and Roll Hall of Fame and Museum – TinEye couldn’t find any similar/matching images
  • Pro Football Hall of Fame interior – no similar/matching images
  • mystery yellow flower – no similar/matching images
  • sunflower macro – no similar/matching images
  • Roz – no similar/matching images
  • Seattle skyline – correctly identified and displayed similar/matching images

Final Thoughts

Both Google’s Search By Image and TinEye performed very similarly with the images that I uploaded from my collection: They did very well for obvious landmarks and things that are commonly photographed and found online, and not very well for other types of images. The Rock Hall building is the only image that Google identified and TinEye didn’t.

But my sample was extremely limited and designed just to get a general sense of how Google’s new feature works, and how it compares to one that’s been available for years now. To get a more accurate sense of the two, you’d need to do probably dozens of more uploads and more detailed testing on whether things like image size matter (my colleague Gary Price thinks it does, based on some of his own tests with Search By Image).

For me, the final verdict is this: If you need to know what an image is and plan to use Google’s Search By Image or TinEye, you better hope there are already a lot of similar images online and, in Google’s case, a lot of web pages (preferably one from Wikipedia) that describe the image.

Postscript: See our follow-up post, Disturbing Michelle Obama Image Makes A Case For Facial Recognition In Google’s New Image Search.

Related Topics: Channel: Consumer | Google: Images | Search Engines: Photo & Image Search | Top News

Sponsored


About The Author: is Editor-In-Chief of Search Engine Land. His news career includes time spent in TV, radio, and print journalism. His web career continues to include a small number of SEO and social media consulting clients, as well as regular speaking engagements at marketing events around the U.S. He recently launched a site dedicated to Google Glass called Glass Almanac and also blogs at Small Business Search Marketing. Matt can be found on Twitter at @MattMcGee and/or on Google Plus. You can read Matt's disclosures on his personal blog.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://midtownseo.com Michael Gold

    When searching for my name, there are some noticeable improvements. The results are nearly all head shots of men named Michael Gold– no women, no animals. On the first page, the only truly inaccurate result is a map from whitepages.com with a description that reads “Michael Gold is most likely to live in Alabama, Arkansas, Arizona.”

  • mvn

    The yellow flower is a Coreopsis.

    Google’s search is really a lot more useful when searching for works of art. Not only you identify the painting, you get a lot of info related to the art.

    Thanks for the nice post.

  • http://campeaux.com/ Brandon

    Your results were way better than mine. When I searched by my avatar, the first image match result was that of Osama Bin Laden. I’m assuming the image matching has more to do with color tone than features. (see: http://twitpic.com/5btwc6 )

    Google Goggles seems to be more accurate entirely.

  • http://emarketingwall.com/ Tharindu Gunawardana

    Is this work only based on similar image recognition or with the help of text. Either file name or description.
    If you are getting more accurate results after you entered the description , what will be the difference compared with Google normal image search?

    I know that this will recognize similar images on what ever technology they used , but how it will enhance the user behavior and help to improve search results ?

  • http://pascal.vanhecke.info Pascal Van Hecke

    Just reading this comparison now.

    I did a few tests with profile images for this blog post:
    http://pascal.vanhecke.info/2011/08/11/reverse-image-search-and-profile-pics-unwanted-linking-of-private-and-public-information-airbnb-as-an-example/

    It turns out that reverse image search does pose some problems for companies that obfuscate users’ full names for privacy reasons, yet allow to have the profile pics indexed. The example I discuss is Airbnb, but also Foursquare comes to mind (where they obfuscate the last name as well).

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide