Video Search Challenge Isn’t Speech Recognition, It’s Content Owner Management

Millions of Videos, and Now a Way to Search Inside Them from the New York Times is a big giant love story to video search firm Blinkx, suggesting that the idea of finding video content will take a leap through new idea of speech recognition. In reality, it’s not a new idea. It’s been in practice for years. And despite those years, it has failed to transform how we search for video on the web. That’s because speech recognition video search is overrated, especially given the true challenge video search faces — just getting the content centralized in the first place.

Search engines cannot really "see" video to understand what it is about any more than they can see images. Instead, to really understand what images or videos are about, they tend to look at metadata — text about the video that is either embedded within the video file or surrounding where the video is placed on a page.

Metadata is a pretty poor way to describe a video. Metadata information tends to be a short summary of what the video is about, who authored it, when it was shot and so on. Anyone who has filled out a basic video submission form when uploading to a service like YouTube is creating this type of summary metadata.

Metadata might only be a few hundred words long, at most. In contrast, the video itself might contain thousands of spoken words. So why not make video search better by capturing those words? That’s the Blinkx pitch — and has been the Blinkx pitch for several years. Despite this, Blinkx has failed to significantly grow in usage. In my view, this is because video search has been less about finding what’s spoken in videos and more about finding what’s hot, something that YouTube and other sharing and rating services excel at.

I’ll do some history to explain this more. But let me give a top level rundown on the main "technology" points to understand about how consumer-facing video search has developed. Services have been based on one or more of the following:

  1. Finding video by reading closed-caption or transcript information
  2. Finding video by crawling the web
  3. Finding video through sharing and rating
  4. Finding video by working with content partners

FYI, for a deeper drill down into some real underlying technology of video search, be sure to check out Niall Kennedy’s The current state of video search from last October.

1998-1999: Closed-Caption & Transcript Searching

Let’s start with the first technology item, that of using closed-caption or transcript information. It’s not at all new. Back in 1998, I wrote about how AltaVista had partnered with Virage to make it possible to search through President Clinton’s video testimony over the Monica Lewinsky affair. Enter a few words, and you were magically taken to the right place in the video. This magic was done through using the closed-caption information:

The service was made possible through software from a company called Virage, and how it was produced in this case was pretty straightforward. The company captured the video testimony and its closed-captions when it was broadcast on CSPAN, the cable network that covers US politics.

The closed-caption text was converted into HTML files, which in turn were associated with 158 video clips. AltaVista then used its search engine technology to index the HTML files, allowing users to find specific text and then view the associated video clip.

"What you saw was very easy for us, because it was closed captioned video," said Dave Girouard, director of product marketing at Virage [FYI, Girouard now works for Google].

Closed caption makes things easy because someone has already transcribed the video tape. Things aren’t so easy when closed-captioning isn’t available, but Virage has ways around this. It can turn to TelePrompTer text or scripts. It has also announced a partnership with IBM for speech recognition, for times when no written record is readily available.

Note the mention of speech recognition. That’s important because not all videos have close caption or transcribed information. That’s why Blinkx says it’s compelling. It will do speech recognition of all videos.

Well, that’s still not new. Speech recognition to create searchable text got a boost when Compaq (later HP) launched the now closed Speechbot service in 1999. It allowed you to do keyword searching against thousands of hours of both audio and video content.

Despite these early technology demonstrations, neither service grew in popularity because of speech-to-text recognition. Indeed, Virage (which powered the AltaVista service) ended up getting purchased by enterprise search company Autonomy in late 2005 (Autonomy, in turn, provides Blinkx with its technology). While I believe Virage continues to be recognized as a leader in the space, in terms of video search by text/speech recognition, it doesn’t provide major consumer facing products. That seems to be because the demand isn’t there.

2003: Crawling For Video Content

Video search largely languished after the initial AltaVista experiments. There was some content, but it was really Singingfish that made a splash when it came along in 2003 and started providing backend video search for Microsoft’s Windows Media Player and for the RealOne Player. By the end of the year, it was also acquired by AOL. That transformed AOL into being the only major search service offering video search at the time (AltaVista, then owned by Yahoo, was no longer a major service).

Singingfish’s main technological claim to fame at the time was crawling to build a large index of content. Both AltaVista and FAST/AllTheWeb also were crawling at that time, but they had much smaller databases.

Singingfish didn’t use closed-caption or transcribed information. Instead, it depended on more restricted metadata. But even though it was restricted data, that plus the larger database of videos made it better than its competitors, when I looked at it back then.

2004-2005: Return Of Crawling, Transcribed Information, Speech Recognition

At the end of 2004, Blinkx launched a standalone site offering video search based on speech recognition. Two years later, it’s that "new" thing that the New York Times reports that we need. So why didn’t it take off in the past two years? As I’ll explain, we needed other things more.

Yahoo Video also kicked off at the end of 2004. Like Singingfish, it went the spidering route. That failed to win Yahoo any major share. By mid-2006, Yahoo shifted to allow video upload and sharing.

In 2005, Google kicked off Google Video. The New York Times article today about Blinkx suggests that Google Video can’t do transcription search:

But search engines — like Google — that were developed during the first, text-based era of the Web do a poor job of searching through this rising sea of video. That’s because they don’t search the videos themselves, but rather things associated with them, including the text of a Web page, the “metadata” that computers use to display or understand pages (like keywords or the semantic tags that describe different content), video-file suffixes (like .mpeg or .avi), or captions or subtitles.

In reality, this is exactly how Google Video started off. It actually had no video. Instead, it taped TV shows off the air, then allowed you to search for and find still photos from certain segments based on the words found in those episodes. It worked pretty well, depending on the closed-caption information.

Despite this, the service failed to gain real popularity. Google eventually dropped the TV grabs (in part, it appears, to help appease big content owners that never felt these were legal) and started showing video submissions. It chose not to crawl the web — the basis of practically everything Google had done in terms of search before this – because it felt submissions would give it better content. But despite submissions, it still didn’t become a video search powerhouse.

Toward the end of 2005, we also got a taste of hype when Truveo (later renamed SearchVideo) came onto the scene. Truveo was to go beyond looking at words on a page surrounding a video to also look at the visual characteristics of the page. I’d tell you more, but that particular angle seems to have disappeared over the years (it used to be here).

Truveo was acquired by AOL in early 2006 and also continues to power Microsoft’s Windows Live Search Video service (compare Microsoft to SearchVideo for the same query, and you can see this. Also, Microsoft has a "reserve" agreement to tap into Blinkx). But didn’t AOL also buy Singingfish? Yes, and they closed that service earlier this month, redirecting to Truveo-powered AOL Video. Despite Truveo’s supposed killer technology, the last stats I saw on video search popularity didn’t have it leaping ahead of others. Those are older, from May 2006, so things may have changed.

2005: Killer Tech, Sharing

After years of video search efforts, both the major players of Google and Yahoo showed that it wasn’t crawling or transcript searching that would be the killer product. Nor did Blinkx prove speech recognition was some killer technology. Truveo also failed to show there was some technology twist to "understanding" video content that would prove a boon.

Instead, YouTube is the player that emerged as the video powerhouse. Kicked off in February 2005, it climbed and climbed in popularity until acquired by Google at the end of last year. What was different? My view of the magic components:

  1. Easy for anyone to watch video without a demand for some damned video plugin or add-on.
  2. Easy for anyone to upload popular commercial content
  3. Easy for anyone to share non-commercial content and make that popular

Until YouTube, I’d say much of the video search assumption had been that people would want things like news or documentaries — and in particular — the ability to search through news content to watch particular things that were said. Instead, it seems the real demand was for video search to be a way to get video on demand and especially to find popular entertainment.

The Lazy Sunday skit (try here to see it) from Saturday Night Live is a classic case in point. Aired at the end of 2005, many flocked to YouTube to see it after it missing it during the live airing.

YouTube, of course, has been notorious for having so much content that might be reposted illegally on the service. But it’s that same content that made it a compelling reason for many to seek it out. In addition to the commercial content, the sharing and voting system there has also allowed non-commercial content to go viral and become popular in a way that previous video search systems didn’t allow.

In particular, it’s not just voting that helped the non-commercial content. It’s the hosting. It’s expensive and a hassle to host video content for small site owners. YouTube made it possible for anyone to effectively become a broadcaster efficiently. Sure, a crawler-based video search service could (and they did) find non-commercial content hosted elsewhere on the web. However, allowing for content uploading fueled a content explosion.

2006/2007: Forget Speech Recognition, Bring On Meta Search

YouTube’s happy days of having commercial content for free are rapidly drawing to a close, of course. This month, we’ve seen Viacom yank its clips off the service to move the content instead to Joost. That goes directly to one of the three magic components of YouTube — the loss of popular commercial content. Now go back to the New York Times article about Blinkx, and its lead about why Blinkx technology is compelling:

The World Wide Web is awash in digital video, but too often we can’t find the videos we want or browse for what we might like.

No we can’t. But that has little to do with not being able to search against spoken text in those videos. The real challenge is that much of the video we want isn’t online or if it is, it is rapidly being removed as content owners put further pressure on sharing sites like YouTube.

The result may be a fragmented video spectrum, where you may have to tune into iTunes for content from one company, YouTube for another, Joost for a third and so on. The solution is actually something else that Blinkx is strong for – not speech recognition but instead meta search. Meta search is the ability to search against a variety of sites and bring back consolidated results from all of them.

To illustrate this, again, a return to the New York Times article. It noted that the aforementioned Lazy Sunday video could be found on Blinkx:

To experiment, I typed in the phrase “Chronic — WHAT — cles of Narnia,” the shout-out in the “Saturday Night Live” digital short called “Lazy Sunday,” a rap parody of two New York slackers. I wanted a phrase that a Web surfer would know more readily than the real title of a video. I also knew that “Lazy Sunday,” for all its cultish fame, would be hard to find: NBC Universal had freely released the rap parody on the Internet after broadcasting it in December 2005, but last month the company insisted that YouTube pull it.

Nonetheless, Blinkx found eight instances of “Lazy Sunday” when I tried it last week. By contrast, Google Video found none. Typing “Lazy Sunday” into the keyword search box on Google’s home page produced hundreds of results — but many were commentaries about the video, and many had nothing to do with “Saturday Night Live.”

Heh. This was overkill in experimenting and the result had nothing to do with speech recognition. First, I think most people seeking the Lazy Sunday video actually would know it more by that name, "lazy sunday." Second, Google Video found no matches because Google Video only searches what’s in Google Video. If the clip has been yanked — as the article itself says it was — there’s nothing to find speech recognition or not.

It’s like saying you couldn’t find a tree in the forest because you lacked tree recognition technology. Sure, tree recognition technology might help. But if someone’s chopped down the tree and dragged it out of the forest, all that technology isn’t going to make it appear.

The reason Blinkx finds the clip is because Blinkx is meta searching. Rather than hosting the content itself, it builds a database of content that is hosted by others from across the web. That helps protect it to some degree from takedown notices.

In particular, here’s the search I did for Lazy Sunday. The top item led me to the video here as hosted by Brightcove, which allows video sharing like YouTube. I can’t tell if Brightcove has a deal to be showing this clip or not (one past New York Times article suggests yes).

In either case, a meta search service like Blinkx wins (and similar services like Search For Video). If YouTube has to drop the video, then the video is gone since YouTube only searches what’s in its own database. But Blinkx, tapping into multiple databases, may continue to list the video.

If Blinkx does get hit with takedown notice, it can remove that link and wait until someone else puts it up elsewhere. Alternatively — and this is most important — by tapping into many video search providers, it defragments the video search market. It becomes a place where anyone can turn to in order to search across multiple video content providers. Of course, the real weakness it faces is if any of these hosting providers decide to block access for Blinkx to index their services.

I’ve covered a variety of search providers in this article. If you’re looking for more, you might check out Video Search Made the Internet Star from last November at Search Engine Watch and Video Search Engines And Online Video Directories: A Mini-Guide from Robin Good. I also guaranteed that Gary Price of ResourceShelf will be along in the comments area below to add his thoughts and resources. Feel free to contribute your own, as well.

Overall, I’m sure speech recognition will eventually find its place in video search. Plus let’s skip the hype. It hasn’t been a crucial piece of technology for years, and it’s hardly the next challenge video search faces. The real challenge is figuring out how to work with content owners.

Related Topics: Channel: Video | Search Engines: Video Search Engines


About The Author: is a Founding Editor of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also serves as Chief Content Officer for Third Door Media, which publishes Search Engine Land and produces the SMX: Search Marketing Expo conference series. He has a personal blog called Daggle (and keeps his disclosures page there). He can be found on Facebook, Google + and microblogs on Twitter as @dannysullivan.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • felixwright

    There appears to be slight confusion in this artictle over meta search, site search and the ‘standard’ search engine search.

    Meta search refers to searches performed across multiple search engines to generate lists of results to return to the user. Examples of this are dogpile, mamma, where there is no single central search index to query against, and calls are made to third party sites as users use the service.

    ‘Standard’ search engines (such as and google) will index and return data from sites other than theirs – but are not meta search engines. The use a logically single search index to return search results to their users, and it is this search index which contains all the information about the sites and pages that can be searched against.

    Site search is of course search restricted to the content on the website where the search appears.

    Youtube and google video primarily fall into the site search category – they will search accross the content hosted on their site only. Any video returned by a search on youtube or google will return only videos hosted by them. Blinkx, and other video search engines such as follow the ‘standard’ search engine model, where searches will return videos from a large number of sites, with searches done against the central index.

    Invoking a particular search result will direct the user to the original site where that data is hosted – (exactly the same model as, and all other ‘standard’ search engines.) Hence Blinkx, or any other video search engine that aggregates videos from around the web and indexes them to a central search engine, are not meta search, they simply follow the same model as and

    These search engines allows users to search for and find data from a wide range of sites, all which have been spidered/captured and stored in the search index.

    The distinction between meta search and ‘standard’ search is an important one. Meta search engines rely on the technology, algorithms of existing search indexes (often with their own proprietary algorithms working to unify multiple search results lists in real time. Hence any item of data not already stored in one of the search indexes will not be returned.

    ‘Standard’ search engines (, do not rely on third party search indexes to return search results in real time. Their own index of the searchable data is used to generate results.

    Each of the two architectures has its benefits, and indeed a combination of the two is often used in an attempt to give users the data they are requesting, but its important not to confuse the two!

  • Danny Sullivan

    I see your point on the meta clarification. Yes, that’s how it works with the major search engines. They each collect pages from across the web, and a meta search engine collect results from each of them (rather than the pages themselves).

    Google Video, YouTube — they don’t scan pages/video from across the web. They are effectively their own sites that someone like Blinkx would crawl as a “real” search engine does.

    But still, I actually think the meta description works better. That’s because while they might centralize the listings, they’re still tapping into services that themselves are primarily places that people search.

    That’s also a core component of what makes meta search — that instead of searching if several different places, you search at one place — irregardless of the centralized database. I’d point someone to Blinkx or Searchforvideo because it would save them from having to search at YouTube and many other places individually. That’s the same reason I might point someone to a regular meta search engine.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide