Wired implies that Kayak is concerned about the similarities causing brand confusion with its users, and even noted an “uncomfortable” comparison in their own review of Bing, and points out that others have noticed as well. You can see further evidence of others comparing Bing vs Kayak from a variety of sources, including TechCrunch’s early review, where commenters pointed out that Bing Travel gave lower airfares than Kayak, though they use the same airline data sources.

Microsoft spokesperson Whitney Burk denied the accusations , telling Wired:

“We are discussing the matter with Kayak,” Burk said in emailed statement. “Bing Travel is based on independent development by Microsoft and Farecast.com, which Microsoft acquired in 2008. Any contrary allegations are without merit.”

However, the Farecast technology which Microsoft acquired is not the issue here, since Kayak does not run a similar fare prediction service, they are not claiming an infringement on technology, but just the user interface, namely Bing’s use of “sliders”.

Sliders have become a popular tool on many “Web 2.0″ based sites, and Farecast did initially use them in design.  Last year, another start-up in airfare comparison, InsideTrip.com, debuted (May 2008) using sliders much like Kayak or Bing does. In Greg Sterling’s write-up, you can see the similarity:

Note that InsideTrip was also founded by Dave Pelter, who held two year executive position at Farecast.com. Is it the fact that InsideTrip.com has not gained (or rather, sustained) enough traction for Kayak users to be as affected by the similar slider options?

Hakia holds similar views

Kayak isn’t the only one crying foul, as hakia also blogged when Bing launched, hakia claimed that Bing’s categorized search is a similar version to its own hakia Galleries, using the example of a search for “Obama”:

“hakia Galleries prove 17 aspects of this query. We save the user time by answering 17 Obama related questions in one search. Compare the hakia Obama gallery with the same search at Bing.com (Bing provides only 7 aspects of this search query).”

Hakia’s COO, Melek Pulatkonak also recently told Investor’s Business Daily that they’d been in early partnership talks with Microsoft in July, and stated:

“We were approached by Microsoft to show them how the hakia galleries worked, and we did, and now they have a similar feature — we showed them how to do it,” she said. “We were surprised that it is a featured part of and the most differentiated part of Bing.”

Microsoft also denies that hakia had any part in the development of Bing’s categorized search, which Microsoft refers to as “faceted search”.

Now the company is introducing a customizable start page (much like iGoogle, Netvibes or MyYahoo) called MyHakia. The company is calling it a “digital newspaper,” which is a good marketing hook.

The hope is that use of this page will expose people to Hakia’s search results.

The Trusted Results program is an initiative Hakia  has developed with information professionals and librarians. As they note on their site:

A popular Web source may not always be credible, and a credible Web source may not always be popular. hakia is the first search engine to integrate librarians’ collective knowledge of credible Web sites into search results to guide searchers.

So far, these results are available for health, medical, and environmental topics and they are looking to expand coverage. Below, more information about what’s new at Hakia, how to vette sites for their new program, and how the changes stack up compared to the rest of the search industry.

Every search engine, whether it’s trying to become the default search experience or provide a niche vertical search, operates within the shadow of Google. Google helped usher in a searching culture, and not only are searchers more likely to”Google” something than they are to go elsewhere to search for something, but the Google experience is the search behavior they tend to expect.

So how can other search engines compete? Certainly distribution and awareness are driving factors in getting searchers to try something new, but user experience is key in getting them to stay.

Building a better search engine

The plan for building a better search engine often includes the following building blocks:

• Comprehensiveness. Cuil stressed this when they launched earlier this year, but they seemed to have lost footing with freshness, which is a key obstacle to overcome when managing so much information.
• Relevance. Google is widely credited with gaining popularity due to its breakthroughs in algorithmically measuring relevance across the web. PageRank was a major factor, but understanding query intent and usefulness of sites based on that query are at least as important to relevance quality as isolated page popularity.
• User experience. Many search engines have experimented with an experience that’s differentiated from Google. While these interfaces may be a step forward for search, the fact that they’re different from Google (which is the behavior searchers expect) makes their adoption more difficult. Ask was praised for its innovative approach to UI with Ask 3D, but as of today, they’ve replaced that interface with what appears to be more like their previous approach.

Hakia’s approach to relevance

Hakia’s initial launch tackled relevance by applying semantic technology. As I noted from my talk with them last year:

They point out that while the traditional search engines bring back good results most of the time, it’s impossible to know if pages that weren’t returned (because they have too few links to them, for instance) would have been more relevant for the query. By understanding the concepts on web pages rather than relying on things like external links and anchor text, they feel they can have a better sense of what page across the entire web is most useful to a searcher.

Human-powered search results

With this latest launch, they’re expanding their focus on relevance by engaging with librarians to manually compile lists of “trusted” sites for particular categories. They say that “Google (and others like Google) don’t make the distinction of what is credible (or quality) what is not.” They note Sir Tim Berners-Lee’s recent comments when launching the World Wide Web Foundation that new systems are needed that give trustworthiness labels to web sites that have been proven to be reliable sources.

Of course, Hakia isn’t the only search engine that’s partially human powered. Mahalo uses both paid and volunteer guides (although they aren’t necessary experts in the subject matter) and even Google itself recently entered the “credible sources” fray with Knol.

Differentiated user experience

Hakia is also launching a new way of navigating search results. The All results tab displays web results, credible sites and news results. Searchers can also view only pages from credible sites, images, or news and have easiest access to galleries.

Many search engines have been experimenting with results beyond the traditional “10 blue links”. However, as of yet, these new experiences haven’t seemed to have pulled searchers away from Google. Google itself, of course, continues to experiment with new experiences as well, including continued evolution of its universal search results as well as more radical changes such as those seen with SearchMash.

It’s great to see search innovation continue and this is just another example of how some part of that innovation is finding the right balance between algorithmic and manual assessment.  I’ve joked lately in talks that searchers see Google as a “truth machine”, when in reality it’s just presenting what it finds with no commentary on its validity (with Knol being an exception). As we rely more on the web, we may have more need for the Berners-Lee vision of trust labels, and search engines like Hakia are providing a glimpse of what that might look like.

Can any start-up search engine “be the next Google?” Many have wondered this, and today’s launch of Cuil (pronounced “cool”) may provide the best test case since Google itself overtook more established search engines. Cuil provides what appears to be a comprehensive index of the web, offers a unique display presentation, and emerges at a time when people might be ready to embrace a quality “underdog” service.

The big questions now are how does the relevancy hold up and can word-of-mouth really still build significant share? [Note: The Cuil site was supposed to be live for searches at of 9:01pm Pacific time on July 27, but so far I'm still seeing only a holding page. I'd expect this to change fairly soon].

Why Care About Cuil?

There’s no end of companies that have been trying to take on Google as a search destination. Earlier this year, my Google Challengers:2008 Edition article covered some of these, like Hakia, Mahalo, and Search Wikia. You can add to that list other companies like Gigaweb and Exalead. None of them have made a dent in Google’s share.

Indeed, the established players of Yahoo, Microsoft, and Ask.com — all of whom have established quality search products — haven’t dented Google either. So what makes Cuil worthy of special attention?

For one, Cuil has an impressive pedigree with its three founders: Tom Costello of IBM’s WebFountain project, plus Anna Patterson and Russell Power of Google’s TeraGoogle project, Google’s massive search index. Cuil also counts former AltaVista founder Louis Monier — who later went to eBay and then Google — as part of the team.

These people know search. In particular, they know on-the-firing line, heavy duty, industrial strength search. Not only that, they’re unleashing what appears to be a comprehensive service that anyone can use. Indeed, Google already did a blog post in reaction to Cuil and its size claims on Friday, before Cuil even launched or those claims became public. If Google’s paying that much attention, then anyone should.

What Cuil Offers

There are four major areas that Cuil is putting out to distinguish itself from other services. These are:

• Big web index
• Unique relevance algorithm
• Unique results display
• Privacy

I’m going to dive into each of these areas in depth, to examine the importance of them as well as dissect some of the misconceptions and PR spin that they also have.

Size Wars Return?

Cuil is claiming to have the largest index of the web: 120 billion pages indexed (with a total of 186 billion seen by its crawler; spam and duplicate content are among things excluded from what gets indexed). In talking with them, Cuil estimated they were three times the size of Google. Sounds pretty awesome, right?

Sigh. Yes, size matters. You want to have a comprehensive collection of documents from across the web. But having a lot of documents doesn’t mean you are most relevant. As I wrote back in September 2005, when Google famously dropped the number of documents it had indexed:

Last century, in December 1995 to be exact, AltaVista burst upon the search engine scene with what was at that time a giant index of 21 million pages, well above rivals that were in the 1 million to 2 million range. The web was growing fast, and the more pages you had, the greater the odds you really were going to find that needle in a haystack. Bigger did to some degree mean better.

That fact wasn’t wasted on the PR folks. Games to seem bigger began in earnest. Lycos would talk about the number of pages it “knew” about, even if these weren’t actually indexed or in any way accessible to searchers through its search engine. That irritated search engine Excite so much that it even posted a page on how to count URLs, as you can see archived here.

While size initially DID mean bigger was better, that soon disappeared when the scale of indexes grew from counting millions of pages to tens of millions. Bigger no longer meant better because for many queries, you could get overwhelmed with matches.

I’ve long played with the needle-in-the-haystack metaphor to explain this. You want to find the needle? You need to have the whole haystack, size proponents will say. But if I dump the entire haystack on your head, can you find the needle then? Just being biggest isn’t good enough.

That’s why I and others have been saying don’t fixate on size for as long as 1997 and 1998. Bigger no longer meant better, regardless of the many size wars that continued to erupt. Remember, Google — when it came to popular attention in 1998 and 1999 — was one of the tiniest search engines at around 20 to 85 million pages. Despite that supposed lack of comprehensiveness, it grew and grew because of the quality of its results.

Why have the size wars persisted? Search engines have seen an index size announcement as a quick, effective way to give the impression they were more relevant. In lieu of a relevancy figure, size figures could be trotted out and the search engine with the biggest bar on the chart wins!

Given this history, seeing Cuil trot out size figures is incredibly disheartening and a step backwards, not forwards. Time better spent on other things (such as measuring the RELEVANCY of the results) will instead get consumed by those trying to count pages. Without even running queries and trying to perform comparison counts, I already have issues with Cuil’s claims. For example:

• Cuil told us that Google was at 40 billion documents. According to? According to what Cuil has heard that reporters have told them they hear from Google. OK, I talk with both Google and reporters that cover them regularly. I’ve never heard this figure put out there. Cuil later added after the initial talk with them that comparison testing makes them believe that Google hasn’t grown.
• Yahoo was said to be at 20 billion. Cuil said this is based on where Yahoo said it was back in 2005, with the assumption that if they’d gotten bigger, they would have announced this. Bad assumption given that since 2005, the search size detente has kept both Google and Yahoo from talking about size figures.
• Microsoft was said to be at 12 billion. Actually, Microsoft said it was at 20 billion last September — but if that hard figure isn’t being used by Cuil, then you start doubting the other ones they’ve put out. In a follow-up, Cuil said they believe Microsoft has fallen back to a smaller index of 12 billion, based on its testing.

We can also start testing in short order, however. Just run a query, see what Google reports as a count for it, then run the same thing on Cuil. If Cuil regularly reports more, they win. Or not. This is what people especially started doing in droves during the last size battle between Google and Yahoo, and then issues about duplicate content and spam starting coming up.

Assuming you get beyond that, any advantage Cuil has on the size front right now will be short-lived, if they make size an issue. Google will simply crawl more documents and ensure that whatever Cuil is, Google will be +1.

We asked Cuil about this, why Google wouldn’t just match them. “If they wanted to triple size of their index, they’d have to triple the size of every server and cluster. It’s not easy or fast,” said Patterson.

In a follow-up, Cuil added that Google being as large as they estimated it to be now was largely down to Patterson’s work at Google, and since she’s no longer there, increasing the index size will be a “non-trivial” exercise.

Perhaps. And perhaps the infrastructure that Cuil has built does make it easier for them to more cheaply index documents from across the web than Google. But Google has plenty of money and engineering expertise of its own. It’s foolish to think they wouldn’t counter what might be perceived as a weakness. They responded to Yahoo in 2005; they’d do the same with Cuil. And for what? Even if Cuil is bigger than Google, it doesn’t mean Cuil is more relevant. Nor does it mean adding more documents in a “I’m bigger than you” game would improve the state of search overall.

Unfortunately, Google started reacting to Cuil’s claims even before Cuil made them. In a post on Friday, Google just so happened to decide it was time to mention they “knew” of 1 trillion items on the web. That will confuse some people into thinking Google has indexed 1 trillion documents, even though they don’t say this. What Google did say clearly was:

We don’t index every one of those trillion pages — many of them are similar to each other, or represent auto-generated content similar to the calendar example that isn’t very useful to searchers. But we’re proud to have the most comprehensive index of any search engine, and our goal always has been to index all the world’s data.

My response to Google — and to Cuil — and to any search engine that tries to do the size battle is what I said on Friday:

There’s no exact answer to what’s a useful page — and so in turn, there’s no one exact answer to who has the “most” of them collected. Tell me you have a good chunk of the web, and I’m fine. But when Google or any search engine start making size claims, my hackles go way up. There are better things to focus on.

As a side note, one issue with any large index is keeping it fresh. Cuil says that they crawl 1 to 1.5 billion pages per day, which means it would take 3 months to refresh everything they’ve currently spidered. However,some important pages are crawled on a weekly basis, they said. That’s good — but Google has pages that can be added in near-real time thanks to its instant layer.

So Long, PageRank?

Cuil is making a big push that it ranks pages by content, rather than popularity. The idea here is to poke at how Google is commonly viewed to just reward pages that have the most PageRank value.

The problem is that PageRank is just part of the way Google ranks pages. It looks at a variety of other factors, so that ranking is not just a popularity contest (see What Is Google PageRank? A Guide For Searchers & Webmasters for more about this).

The other issue is that despite the PR pitch, Cuil is indeed using popularity to rank results, as far as I can tell.

For example, in a search for [harry potter], the Harry Potter & The Order Of The Phoenix movie web site comes up first on Cuil. This is out of thousands of possible pages. How on earth can Cuil know just from the content on the page itself that the movie site should be in the top results, especially in a web environment where people can (and will) custom tailor content to mislead search algorithms?

The answer is link analysis — counting links and effectively seeing who is pointed at the most. The twist is that it is done by measuring the links from pages relevant to what someone search on.

Let’s go back to the [harry potter] search. When you do that at Cuil, it finds all the pages that it thinks are related to those two words. This means pages that use those words, as well as pages that have other words on them, such as “harry potter books” or “gryffindor.” It figures out these relationships by seeing what type of words commonly appear across the entire set of pages it finds. Since “gryffindor” appears often on pages that also say “harry potter,” it can tell these two words (well, three words– but two different query terms) are related.

Cuil then looks at the entire set to see which pages are linked to from them. Those with many or important links are likely to do better. Since the Harry Potter movie page has a lot of links pointing at it, it comes up higher in the results. Cuil even has a name for this — IdeaRank.

If this sounds familiar to some people, that’s because this particular flavor of link analysis was popularized by Teoma, which was later acquired by Ask. When Teoma appeared, it tried to distinguish itself against Google by saying it analyzed only the “subject-specific” links to do ranking. This is still played up at Ask today:

Our ExpertRank algorithm goes beyond mere link popularity (which ranks pages based on the sheer volume of links pointing to a particular page) to determine popularity among pages considered to be experts on the topic of your search. This is known as subject-specific popularity. Identifying topics (also known as “clusters”), the experts on those topics, and the popularity of millions of pages amongst those experts — at the exact moment your search query is conducted — requires many additional calculations that other search engines do not perform. The result is world-class relevance that often offers a unique editorial flavor compared to other search engines.

Fair to say, despite Ask’s supposed improved analysis, it never trounced Google. Moreover, there are plenty who assume –including myself– that Google itself does subject-specific link analysis.

So the rank by content twist at Cuil? As I’ve said, more twist than substance. But the content analysis is used in other ways, as I’ll get into next.

Three Column “Magazine” Display

Probably the most dramatic difference between Cuil and Google is how Cuil runs search results across three columns, rather than all in a straight line:

It’s appealing in one sense that you can see more results all that once. In fact, Cuil said in user testing, the display had an impact on which results was seen as “number one.” Some viewed the result in the top left corner as most important — others go to the one in the top left. When all nine results can be seen on a large screen, some assume the one in’ the middle is best.

In the top right corner, there’s a “Related Searches” box that allows you to refine your search and drill into specific topics:

Vivisimo a long-time pioneer and Yahoo recently tapping into this to some degreewith its Yahoo Search Assist service.

Somewhat related to this, “tabs” appear at the top of the search page listing related searches:

Select one of these tabs, and you get back results on that particular topic. Tabs that appear reflect how popular those phrases are on the web.

Underneath the display, Cuil is also working to do what it views as a twist. It’s trying to diversify the results by topic, so that in the case of Harry Potter, for example, you might get pages about the book as well as the movie and the author, rather than just results that are all about the book.

“We’re trying to choose pages that aren’t all on the same topic and show you the diversity of the web,” said Patterson.

Again, however, it’s not like a Google search lacks diversity. A search on Harry Potter there also brings back different types of results.

Actually, a far more dramatic example of results diversity is what someone like Hakia does. Unlike Cuil, which divides pages into topics based on word patterns, Hakia is doing real semantic analysis — trying to understand what words actually mean and what pages are about. As a result, it groups results for [harry potter] into various categories such as:

• The Books
• Headlines
• News & Interviews
• The Soundtrack
• The Games
• Photographs & Pictures
• Blogs & Fan Sites
• Myths & Controversies
• Merchandise

Over at Mahalo, you get similar groupings — though these are done through human work rather than through concept analysis.

Where are the ads? At launch, there will be some public service ads at the bottom of the page, so that people are used to ads being in that spot. As for revenue generating ones, the company is considering whether it should build its own ad network or partner with someone else.

Related to display, Cuil automatically suggests search topics as you type into the box on its home page, queries that come from looking at the most popular related words from across the web. It will also suggest actual web sites to take you to, showing an icon next to their name:

The Privacy Card

Cuil says that it’s not logging IP information or keeping any type of material that could be traced back to individual searchers. In contrast, all three major search engines do log IPs addresses plus cookie searches and, in the case of Google, even allow searchers to store search history over time.

That may be reassuring to some searchers, but to date, even scare stories about what Google could do (not that it does) hasn’t kept searchers away from it. Ask and Microsoft have both tried to play the privacy card against Google and gained nothing for it. Small player Ixquick has been playing the card even longer and has gotten no visible traction out of it.

Another issue is that by not allowing searchers to voluntarily allow for personalized results, Cuil might be missing out on an advancement in search where Google’s ahead of the pack.

Under The Hood

Behind the scenes, Cuil talk about its infrastructure that’s designed to be faster and more efficient than those at Google or the competing search engines. Supposedly, that means Cuil can do things cheaper and better than the other players.

I’ll leave this to others who are tech heads to dissect more in the future. For my part, I’ll just say that I’ve heard this line many times over the years. AltaVista would say how it was better in architecture than Lycos. Inktomi would talk then about how it was more distributed and cheaper than AltaVista. Then FAST would say it was even more distributed than Inktomi. And Google, you know, was so with it that you could break circuit boards and things just kept working.

I’d also get briefed on how super-duper the Google infrastructure was, for example, then a few years later, there would be a completely new one introduced that lets them do things not possible under the old one.

So call me jaded. Everyone’s always saying they’ve got the best, fastest and least expensive way of doing stuff. If they do, proving it has been difficult, nor has it seemed to make much different in market share.

I’ll leave with a few stats, however. Right now Cuil runs off of two data centers, using a combined 1,000 machines each running 8 CPUs. Another 280 machines split between data centers are used to serve results rather than index the web.

The Name

Does something seem missing with Cuil’s name? They lost the second L that was part of their original name, Cuill.

“We had a moment of silence for the departing L,” said Patterson, explaining that people found it easier to remember the name with one L rather than two.

The name mean “wisdom” in Gaelic, from the legend of Finn MacCuil. That brings in another Teoma connection. Teoma is apparently a Gaelic word for “cunning.”

Will Cuil Succeed?

Does Cuil think it can beat Google in the search space? They won’t come out and say that, but you can’t help but feel that’s the goal when talking to them. How about beating at least Microsoft or Yahoo?

“We’ll take that for a start. If we do that in the next year and a half, I’ll be an extremely happy person,” Patterson said.

But is even that realistic? Microsoft and Yahoo themselves both have mature search products. What’s especially important is that the big three also offer more than the web search that Cuil is providing at launch. News search, image search, video search, local search — these are just some of the verticals that Cuil lacks but which do get used by searchers. Not offering these makes Cuil feel too focused on what “old school” search used to be and missing out on the Search 3.0 vertical and blended search revolution that has been going on.

Clustering of pages; subject-specific link analysis — these are things others have tried and gained no market share with. Having a comprehensive index is great, but no one prior to launch has been able to play with the service and measure core relevancy. Cuil itself said it had no metrics to show it is more relevant than Google. So why would anyone think it could gain share?

I tend to think Cuil’s hitting the timing right to pick up a little share, maybe a point or two (which is huge compared to other start-ups). I don’t think word-of-mouth is dead, and I’m cautiously optimistic that Cuil will have good relevancy even without having tried it yet (if not, all bets are off). I think you’ve still got a core of early adopters and tech geeks out on the web who want the “next Google,” especially at a time with the existing Google seems so big and threatening to others. A good underdog can fill a need, if it’s a quality underdog — and neither Yahoo or Microsoft have that underdog spirit.

It’s possible that Cuil could be a wild success that eclipses Yahoo and Microsoft and does threaten Google itself, of course. Anything’s possible. But I think that’s unlikely for reasons I wrote before:

Google came along at a very special time, as I’ve long written. It had better technology at a time when all the search engines had abandoned improving search, since that was seen as a loss leader. The money was in portal features.

Today, search is a multi-billion dollar industry. If someone with a serious search threat comes along, you buy them (such as with YouTube), or you start to develop your own rival if it seems a real threat. Google’s not omnipotent — but you’ve already got a space where it’s Google, Yahoo, Microsoft, and Ask all seriously fighting it out (and the latter three, despite their funding and experience, still struggle against Google as being synonymous as a trusted search brand for most users).

To date, Google is the real exception of “a better mousetrap wins.” It’s far more likely the companies above, if they do gain traction, will end up being purchased for a large amount by one of the existing “search utility companies.”

Is Cuil open to being purchased, as what happened to search start-up Powerset earlier this month? Powerset was seen by many as a Google-killer (though not by me and several others). In the end, despite the hype that Powerset itself helped fuel (Cuil’s been careful to avoid this), it got gobbled up. What if Microsoft’s Steve Ballmer came knocking?

“I don’t know what he’d be knocking for, whether it be acquisition or partnership or whatever. We do intend on being polite. We believe in getting to know people and making friends because you never know what deal may come down the line,” Patterson said.

For related discussion, see Techmeme.

Postscript: See Cuil Fast Test – Relevancy Isn’t A Google Killer.

By the time you read this, the Powerset site should have changed into a tool that allows you search against material within Wikipedia. Why bother using Powerset rather than using Wikipedia’s own search tool or even Google set to look only within Wikipedia pages? The Powerset pitch is that you’ll get better results because Powerset’s technology has read and understood what every word within Wikipedia actually means.

An Understanding Engine, Not Natural Language Search

To understand that more, I beg that you forget you ever heard "natural language" being associated with Powerset. That’s not really describing what they do in comparison to regular search engines.

To explain, you have to understand that Google and the other major search engines are largely stupid. They don’t really understand the content on the pages that they "read." If they see the word "walk" in a sentence, they don’t know if walk is being used as a verb or a noun. In very general terms, they don’t even know that words are words. Words are more or less patterns to them — collections of letters — and when someone searches, they try to find the pages that have those patterns in them or in links to those pages.

That’s VERY simplified, OK? The major search engines DO have some smarts, some ability to know that walk is related to walking or that walk and run might be similar words. But this is largely done through statistical guessing, rather than comprehending what the individual words actually mean, especially in terms of their exact grammatical usage.

Powerset is different. It says that its technology reads and comprehends each word on a page. It looks at each sentence. It understand the words in each sentence and how they related to each other. It works out what that sentence really means, all the facts that are being presented. This means it knows what any page is really about.

In lieu of a better phrase, call it an "understanding engine." Maybe that’s not the right phrase, but natural language search isn’t it, either. Understanding engines at least highlights the uniqueness of Powerset — that’s because it actually understands what pages are about — it can extract facts from those pages plus comprehend how those facts, as well as those pages, relate to each other.

Wikipedia Discovery Tool

One of the chief uses for Powerset is employing it as a Wikipedia discovery or query refinement tool. To use the Powerset example they gave me during a briefing last week, consider a search for [henry viii]. What’s someone interested in in when they search on that topic, given Henry did a lot of things during his reign?

Over at Google, we get query refinement suggestions at the bottom of the page, like this:

At Yahoo

At Microsoft

Most of these are generated by looking at the relationships between those who have searched for one topic and then may have gone off and done another search. Yahoo has the most sophisticated of the pack (see Search Suggestions On Steroids: Yahoo Search Assist), but it still hasn’t actually "read" about Henry VIII and tried to group him into subtopics, in the way a human might.

That’s what Powerset tries. Here’s what you get in a search for Henry VIII:

Notice the tabs at the top, where it recognizes Henry VIII could refer to the person, the opera, the play, or even a television drama. OK, so not too amazing when you think about it. But look further to the "Factz" area. Here you can see that Powerset, after reading through Wikipedia, has figured out that Henry VIII "dissolved" things like monasteries or that he "granted" things like land. And yes, he "married" a few people.

There’s even more facts that can be found like this:

This is nice refinement. Running down the list, you can quickly scan the many facts that define Henry’s life. And from the list, with a click, you can drill in more about topics and jump right to particular pages within Wikipedia:

See how there’s a link to the Falmouth, Cornwall page? Powerset has seen that there’s something Henry VIII built mentioned on that page, Pendennis Castle. That’s not covered on the main Henry VIII page, but because Powerset has read both pages and understands what they are about, it can link the facts together.

Overkill For Now?

In short, the refinement is cool. What’s not to love about it? For one, it might be overkill. During the demo, Powerset made a big deal on how Powerset could build information from across various Wikipedia pages that isn’t written on any single one of them. For example, a search for [hulk hogan] brought this up:

See how those who Hulk Hogan has defeated are itemized? It’s nice — but do you really trust that all the defeats have been captured? I wouldn’t. I’d probably still go looking for an authoritative list that had been reviewed by a human. Moreover, I can get lists like that without great refinement. A search for hulk hogan victories on Google brings me to this nice page on About.com listing his world title victories.

In addition, while Powerset did a nice job of breaking down Henry VIII according to Wikipedia, Wikipedia’s human editors do a pretty nice job right in the opening paragraphs to the Henry VIII page:

Henry VIII (28 June 1491 – 28 January 1547) was King of England and Lord of Ireland, later King of Ireland and claimant to the Kingdom of France, from 21 April 1509 until his death. Henry was the second monarch of the House of Tudor, succeeding his father, Henry VII.

Henry VIII was a significant figure in the history of the English monarchy. Although in the first parts of his reign he energetically suppressed the Reformation of the Anglican Church, which had been building steam since John Wycliffe of the fourteenth century, he is more often known for his ecclesiastical struggles with Rome. These struggles ultimately led to him separating the Anglican Church from Roman authority, the Dissolution of the Monasteries, and establishing the English monarch as the Supreme Head of the Church of England. Although some claim he became a Protestant on his death-bed, he advocated Catholic ceremony and doctrine throughout his life; royal backing of the English Reformation was left to his heirs, Edward VI and Elizabeth I. Henry also oversaw the legal union of England and Wales (see Laws in Wales Acts 1535–1542). He is noted in popular culture for being married six times.

I suspect most people hitting Wikipedia are already going to find an opening paragraph like that, which does a pretty good job guiding them in refining their topics about Henry VIII and pointing them to facts.

That’s a problem for Powerset, which told me it hopes to  attract lots of those Wikipedia users to its own site, where they’ll be eventually shown ads alongside the content (ads aren’t present at launch).

Powerset was at pains to explain how popular Wikipedia is and what a well used resource it has become. Agreed — and plenty of those people wind up there because they’ve done searches at Google. About 70 percent of Wikipedia users come via search engines, according to Powerset itself. That’s a huge audience that is NOT going to magically be routed to Powerset instead. Yes, some know to go directly to Wikipedia. No doubt some of these users will hear of the new Powerset tool and go there. However, it will be a stunning achievement if these are more than a fraction of those who hit the main Wikipedia site.

Article Outlines

Powerset has another trick up its sleeve that might pull in the people. For any page you visit, there’s an "Article Outline" box that appears within it, like this:

It’s very slick. Select an item, and you’re jumped to the spot within the document related to it:

I think it’s self-evident that Powerset adds some nice value to Wikipedia. Indeed, everyone would probably be smart to go to it directly rather than Wikipedia itself. But as I’ve covered above, that’s not what I expect to happen.

Future In Site Search?

If Powerset fails to capture a wide audience, then what’s the way forward for it? One area is to provide better site-specific searching. Powerset’s technology can be applied to any set of documents, to make it easier for people to find what they are looking for within them. Site specific search allows those visiting a particular web site to look just within that site. That market, along with enterprise search (making intranets searchable) continues to grow. And the audience doing those types of search are likely more inclined to seek out refinement options and other exploratory tools than they are when performing general searches.

Powerset said this is a market they’re interested in, so perhaps we’ll see it grow in that area. But for those expecting it to produce Google-wealth, keep in mind that long-time and mature enterprise search player FAST sold for $1.2 billion earlier this year. Yes, that’s a huge amount of money, but it’s not the multibillions Yahoo was going to go for, and it’s much less than what Google’s valued at.

Speaking of Yahoo, it used to be the leading candidate in the past of who might acquire Powerset, especially given some close ties between the companies (Powerset has a number of former Yahoos on staff). Given Yahoo’s current troubles and unstable state, I wouldn’t expect much here.

Could a tie-up with a major player like Google or Microsoft happen? Sure. Aside from site search, the technology that allows machines to automatically comprehend what text documents are about ought to have other applications and be worth something. What those are and how much it is worth isn’t clear. Powerset’s been smart to snap up many licenses and patents around the technology that should make it attractive to a larger search player like Google or Microsoft to acquire. Within one of these organizations, I suspect more innovative things would come.

FYI, I wrote the above paragraph last Friday, before the rumors (see here on News.com and here on Techmeme) that Microsoft might want to buy it came out over the weekend. Actually, I started writing this article several months ago and in that, was covering how it might be an acquisition target. It’s a fairly obvious move to expect any of the majors to take a look, and when I talked to Powerset several months ago, I was given the impression that all the majors had taken a look.

Since then, of course, no one has acquired it — plus the company went through a management shake-up last year. It was already under fire for not getting a product out for so long. Add to these strikes as a potential Google killer the fact that it takes Powerset about a month to comprehend Wikipedia’s 2.5 million topic pages. In that time, many of those pages will have changed — thus needing to be reread again. Powerset’s impressive, but with the web having in excess of 20 BILLION constantly change pages, this is no overnight secret weapon that Microsoft might buy and employ to take the search lead.

Indeed, what Powerset says it has developed — along with patents locked up to protect it — is overkill for what’s needed today. It will be more useful probably five years from now, in ways we’re not even envisioning. For those players thinking long-term, which include both Google and Microsoft, sure — it might well make sense to buy.

By the way, the Powerset launch will no doubt inspire interest in another "natural language" search engine, Hakia. Someday I want to revisit Hakia and explain more about why I also dislike the term "natural language" being applied to it. In the meantime, you can read Vanessa Fox’s excellent article from last October on the service, Social Networking Through Search: Hakia Helps You Meet Others. And if you need a deflation of natural language hype, see The Google Challengers: 2008 Edition. In the section on Powerset, I summarize a long rant I did on the history and hype of natural language search.

For related discussion, see Techmeme.

The company says that it will roll out “Quality Search” in a range of verticals — “law, finance, science, and in many other content-rich verticals” — based upon expert sources and librarian-aided indexing.

Stepping back, the irony here is that it’s a bit of a return to the “directory” approach of old. At the highest level, humans originally compiled lists of websites (e.g., the original Yahoo directory). That was eventually replaced by machine algorithms when the internet got to be too large to categorize everything with an editorial staff. Enter Google.

But when the internet became so large that information overload became somewhat overwhelming and routine, the trend of human-powered search or “social search” emerged to rectify some of the seeming randomness and inefficiency of these giant indexes. Social search, generally speaking, thus sought to inject a community filter into search results (e.g., Eurekster).

Hakia’s blog post and approach suggests a return to the “top-down” editorial efforts of the earlier days, albeit with the knowledge base and capabilities of today’s internet. (This simplification probably doesn’t fully capture what they’re doing.)

One can also see the blended/universal search trend as an effort to get to “answers” beyond the information overwhelm that intrudes into the prior “10 blue links” approach that defined general web search for years.

Blekko

What we know so far about Blekko isn’t much, and TechCrunch has the most details in its The Next Google Search Challenger: Blekko post from yesterday. Apparently Rich founded the company in September 2006, along with five other former Topix employees, after he left Topix in June.

Rich told TechCrunch not to likely expect anything public until 2009. I agree with Michael Arrington at TechCrunch that Rich has a track record that makes him well worth watching. The Open Directory was an initial success, though the model didn’t scale well. Some of that was within the founders’ control but had more to do with AOL’s lack of backing. The company should be dragged into the International Court Of Search Crimes and be forced to sell the ODP to someone who will support it properly. Topix has built a reputation and is still standing and succeeding — though I’d say it still has far to go to seriously threaten Google or Yahoo.

Rich adds a bit more in his Why Search? post today:

Having just spent 5 years in the media space, I’ve come away with the idea that editorial differentiation is possible. But the editorial voice of a search engine is in the index…so it has to be algorithmic editorial differentiation.

So far, it doesn’t sound like a social networking play like some of the others. We’ll be watching, Rich. Also see discussion today on Techmeme.

Powerset

Powerset is now a classic example of why you WANT to be a stealth start-up and say little. That’s because when you get too much early press — in part through your own doing — then fail to deliver anything, the hype can swing back at you hard.

The company came to light back in October 2006 via VentureBeat, with the twist being that natural language search would be the way forward. That caused me to write a long rant about the hype of natural language search in reaction. From the top of that:

This is a rant. It’s a rant from over 10 years of watching people trot out natural language search as the “killer” solution to the current state of search, something that’s happening once again with Powerset. That’s a search engine you can’t even use at the moment, but the hype will no doubt continue. To counteract that, my thoughts on and some history about natural language search.

Natural language search makes a compelling pitch for those who really don’t know search or haven’t heard the natural language mantra before. I’ve seen the pitch time and time again. You:

• Pick out an example that shows how “bad” search is on an existing search engine
• Demonstrate how natural language search would work better on your service
• Sit back and collect the press attention

I then went on to detail how natural language search had been hyped and tried over the years. The short story is this: It doesn’t take much natural language analysis to figure out what someone wants when they type in “britney spears nude” or “hotmail.” In addition, by and large I don’t believe enough people will change their basic search habits to enter long sentences when searching any time soon.

Since that time, we’ve pretty much had nothing out of Powerset other than the launch of Powerset Labs in September 2007. That launch hasn’t produced any cool applications that I’ve seen or heard about, nor much buzz. Instead, in November, we got a management shake-up.

For a more formal chronicle of the company’s developments, check out this area at VentureBeat and these search results at TechCrunch.

Finally, while I’m harsh above on Powerset, I actually had a long visit with the company in the middle of last year and was deeply impressed with the effort going on there. I’m still working on a long write-up to explain what’s happening. But in a nutshell, Powerset is trying to literally comprehend or understand each page on the web.

Today’s search engines don’t know what a page is about by reading words. They’re more or less doing pattern matching — finding pages that contain words similar to what you search for (or pages relevant to those words based on linkage). Powerset literally is trying to read and understand what a page is about the way a human reads a page and knows it is on various subjects.

I don’t see that as making it a better search engine that Google. Instead, I think it may eventually give it the ability to create a unique “auto-Wikipedia” style site, assembling knowledge pages on any subject automatically. I also think that there will eventually be some search benefit in comprehension of pages, but exactly how that will play out I suspect is part of being with an existing search engine and a more traditional model. With the array of patents Powerset has lined up, I suspect it will eventually get acquired by Google, Yahoo, or Microsoft rather than rollout its own product. But we’ll see.

Hakia

Like Powerset, Hakia has played the natural language search game. Unlike Powerset, it has a product anyone can use — live since at least the middle of 2006.

Again, I’ve been working on a long write-up on the inner workings of Hakia and have yet to finish it. It’s complicated, and I mainly want to cover what I find to be the real use of their technology — the ability to create custom “gallery” pages and understand those are related to particular searches.

It’s easier to show you what’s impressive. Search for hillary clinton, and you get a nice page showing news, her official site, biography pages, blogs & fan sites, news & interviews, and more. It’s very Mahalo-like, except it doesn’t require human editors like Mahalo and predates Mahalo by a year.

That categorization is something I know the major search engines could do, if they wanted. So far, they don’t. And so far, despite Hakia talking about its rising traffic, it has yet to make a serious mark. Moreover, in October, it made a serious shift to allow social interaction with its results. That’s a sign that the original plan that “natural language will win all” has failed to do so; therefore, another twist is needed.

Social Networking Through Search: Hakia Helps You Meet Others from Vanessa Fox here at Search Engine Land covers the change, plus it gets into the natural language indexing stuff I mentioned earlier that makes Hakia unique, plus has examples of gallery pages.

Mahalo

Credit to Jason Calacanis. He said he wanted to take on Google, then wasted no time getting Mahalo rolled out. OK, he also says he’s not taking on Google — just focusing on the top searches that he thinks would be better with human review. Sure, you aren’t taking on Google, Jason.

To date, Jason reports that Mahalo’s traffic is growing and strong. But to date, I’ve certainly see no webmasters taking about what a traffic driver Mahalo is. It would be early to call it a raging success, but it’s a nice alternative to have. Indeed, later this month I’ll finally finish my Search 4.0 piece that picks up from the conclusion of my Search 3.0: The Blended & Vertical Search Revolution article last November. I’ll show some examples of how the human element at Mahalo can and has kicked some Google and traditional search engine butt — though also how it isn’t the panacea some expect.

Some of our past coverage of Mahalo:

• Mahalo Launches With Human-Crafted Search Results
• Mahalo Greenhouse: Get Paid For Writing Search Results
• Search Spam Fight - Mahalo: 1; Squidoo: 0
• Mahalo Follow: Toolbar Gives You Human-Powered Alternatives To Searching, Surfing
• The Promise & Reality Of Mixing The Social Graph With Search Engines
• Mahalo Adds The Social Graph To Search

Search Wikia / Wikia Search

Wikipedia founder (as he prefers to be called; Wikipedia itself calls him cofounder) Jimmy Wales made waves a year ago when he said he’d take on “closed” Google with humans and a transparent search engine. Called Search Wikia (but, confusingly, it’s also called Wikia Search), Wales has grabbed attention from the press over the past year. Slamming at Google as a scary closed thing gets you good mileage, especially when you helped establish Wikipedia, a threat Google takes so seriously that it may launch its own Wikipedia-style site, Google Knol.

Now Wikia Search is at hand. A private “pre-alpha” test started in late December, an invite-only thing I still find odd for a service that’s supposedly all about the “transparency.” But on Monday, the general public will finally get a look at whatever Wales and his team have concocted. In the meantime, while Wales still hasn’t posted any news since July 27 to the “news” section of Search Wikia, press reports tell us so far:

• Only a tiny 50 to 100 million pages will be indexed at launch. The major search engines today have tens of billions of pages indexed. (AP)
• There will be a high degree of human editorial influence, though whether that’s over the algorithm or the search results on a per-query basis remains to be seen (CMP)
• An early screenshot suggested that Search Wikia might be evolving more into a Facebook-style service, perhaps with some ways for users to share results (Matthew Buckland & Wired)

Some of our past coverage of Search Wikia:

• Q&A With Jimmy Wales On Search Wikia
• Search Wikia Takes Steps To Crawl; Acquires Grub
• Search Wikia Gets Open Source Categorization Software
• Search Wikia Launches In 2007 With Private Beta

Cuill

Arguably the stealthiest of the stealth start-ups, Cuill (pronounced “cool”) has an impressive pedigree with its three founders: Tom Costello of IBM’s WebFountain project and Anna Patterson and Russell Power of Google’s TeraGoogle project, its massive search index. And last year, former AltaVista founder Louis Monier — who later went to eBay as its first eBay Fellow, then to Google — jumped ship from Google to join Cuill.

I talked with Cuill earlier this year to understand a bit more about what they are doing, but the details are still being held very closely. The main difference between Cuill and everyone else I’ve named above is that Cuill is founded by people who understand and have dealt with firsthand the challenge of indexing billions of documents.

Cuill recently took on more funding. Louis is also going to be doing a keynote at our SMX West search marketing conference, held in Santa Clara, California from Feb. 26-28. I’m thrilled to be having him since there are only a handful of people who have worked for the “old” Google (AltaVista), the current Google (when he was at the Big G), and a potential future Google (Cuill).

And The Winner Is…

If you think the future of search is on smart automation, Cuill’s definitely one to watch, and perhaps Blekko as well. If you think it’s the growth of humans, Mahalo and Search Wikia are your better candidates. The reality is that success will likely be a blend of the two. For the human services, a real open source index would be a big help — see Google: As Open As It Wants To Be (i.e., When It’s Convenient) for more about this.

But the reality is that all of these services will have an incredibly tough time to beat Google.

Google came along at a very special time, as I’ve long written. It had better technology at a time when all the search engines had abandoned improving search, since that was seen as a loss leader. The money was in portal features.

Today, search is a multi-billion dollar industry. If someone with a serious search threat comes along, you buy them (such as with YouTube), or you start to develop your own rival if it seems a real threat. Google’s not omnipotent — but you’ve already got a space where it’s Google, Yahoo, Microsoft, and Ask all seriously fighting it out (and the latter three, despite their funding and experience, still struggle against Google as being synonymous as a trusted search brand for most users).

To date, Google is the real exception of “a better mousetrap wins.” It’s far more likely the companies above, if they do gain traction, will end up being purchased for a large amount by one of the existing “search utility companies.”

Several companies have been taking the natural language angle in creating the next generation search engine, but turning the potential into production has proven tricky. Powerset had hoped to launch their search engine this year but now thinks it could be as late as 2009 (although they are hoping for a mid-2008 launch).

Hakia, another player in the space, hopes to have their own brand of natural language processing fully powering their search engine next year.

Pell is optimistic about Powerset’s future. In his blog post, he says he’d love to talk to potential CEOs and closes by saying:

“The changes we are making now position us for a next phase that promises to be really exciting. We will bring our technology out in real products that users will enjoy and that will trigger changes across the entire ecosystem of search. I think the next year is going to be an amazing time for Powerset and I am as passionate as ever about Powerset, our technology, our team and our future.”

Since Hakia processes queries differently than old school search engines such as Google, you aren’t just matched up with people who typed in the exact query you did — you’re matched with a larger set of searchers that Hakia thinks are looking for the same things you are based on natural language processing. For instance, if you’re searching for “what drug treats a headache,” Hakia processes the semantic relationships between words and may deduce that someone searching for “what medicine relieves migraines” is a match. And that type of processing is the crux of how Hakia wants to differentiate itself.

I recently sat down with President and COO Melek Pulatkonak and CEO Dr. Riza Berkan to talk about what they’re doing in the search space and where they see things heading. More on how they’re tackling natural language processing below. First, a run down of what was launched today.

In addition to working on a completely new way of indexing and ranking the web behind the scenes, Hakia is also working on providing a unique search experience. In July, they launched the Hakia ScoopBar, which highlights the sections of the pages that are relevant for your query when you click through to them from the search results.

Today, with Meet Others, they hope to add a social networking component to search. The feature is entirely opt in. Once you do a search on Hakia, you’ll see a Meet Others icon above the search results. Click that to access a room designed for those doing similar searches. You can post a message and then provide details about how you want to be contacted (masked email or instant messaging via MSN or Skype). You can also contact others who have posted messages to the room. The freshest and most highly rated posts stay in the room longest. Older and less popular posts fall off as searchers make new posts. Hakia says they monitor abuse and have safeguards in place for spam (for instance, your post is authenticated through email).

What about privacy? Since the feature is opt-in, no one will see what you are searching for unless you decide to click the Meet Others icon and post a message. Even then, no registration is involved so your post isn’t associated with a username. And you decide what contact information you want to make available to others. If you choose email, Hakia masks it so anyone contacting you doesn’t see the address. (However, your IM details, if you choose to post them, are public.) You can also remove your post at any time, which removes any contact information you’ve made available.

Hakia likens this system a bit to craigslist.org. They want to bring people together. They use the example of someone looking for concert tickets. Someone else may post a message about tickets available for sale. Hakia can bring the buyer and seller together.

Here’s social networking search in action. Doing a search for “Looking for collectible pokemon cards” brings up results that include a Meet Others icon to the right of the search button.

Click that to see who has posted about that query.

Then post a message of your own or choose a contact option for someone who’s already posted.

As noted above, there’s no set time limit for how long a message remains in the room. It varies depending on how many messages get posted and how they get rated.

Is this an innovation in search or a recipe for disaster? How many people will find this valuable and how many will find it just plain creepy? In today’s climate of extreme social networking, Hakia just might be onto something, but the proof will be in the adoption.

Getting searchers comfortable with the notion of chatting with others about their searches isn’t Hakia’s only adoption obstacle. As I said in my review of one of Hakia’s CDs (they’re talented musicians in addition to computer scientists, who sing about zebras and finding your childhood on eBay for twenty five cents), regardless of how revolutionary their technology may be, the big challenge will be getting people to change the way they search because their technology isn’t at its best for the 2.8 word queries that Google has taught the world to type in. With those types of queries, Hakia performs just like any other search engine. Their differentiation comes with natural language processing, best used for longer queries that are typed more like the way people talk or write.

And what of this differentiation? Hakia provides interesting results now, but the jury’s still out on just how different and valuable what they are working on really is. They aren’t launching the search experience that’s powered by the core technology they are working on until sometime next year. They say what’s currently launched leverages the technologies they’re developing, so you can get a sense of what the final product will be like.

They say they are working on an entirely new infrastructure (different than what traditional search engines employ) called QDEXing (query detection and extraction). They search the web for concepts, rather than words, when satisfying a search. They point out that while the traditional search engines bring back good results most of the time, it’s impossible to know if pages that weren’t returned (because they have too few links to them, for instance) would have been more relevant for the query. By understanding the concepts on web pages rather than relying on things like external links and anchor text, they feel they can have a better sense of what page across the entire web is most useful to a searcher.

Traditional search engines use inverse indexing to catalog the words on a page. At the simplest level, when someone does a search, the engine looks through the index and finds the pages listed for those query words. Hakia, instead, uses QDEXing to determine what questions each page can answer. When someone does a search, Hakia finds that question and then returns the pages that answer it.

They use a number of scoring factors (such as linguistic and referential methods) to determine page quality. For instance, they say they can detect if a page was written by a person or was autogenerated. They only store pages that they feel meet their quality bar.

So, how will they get searchers to try it out? They may start with vertical databases that don’t do as well with traditional search technology. And they’ve cataloged the 700,000 most popular search results into galleries, which are algorithmically generated and have human review. Groups of hand-edited results for popular queries? Haven’t I heard this story before somewhere? Hakia says they’re different from sites like Mahalo and Wikipedia in that the results are algorithmically generated, so there’s less of an editorial component. They provide a well-balanced representation of query results — not just the most popular. And most importantly, the content is search results, not reference material. Everything about what Hakia does is about improving the search. What does a gallery look like? Well, I randomly chose a query — “Buffy the Vampire Slayer”. No reason. Hakia returns a gallery page with things like headlines, television profile, the channel it’s on, pictures, and fan sites. Hakia algorithmically generates the categories based on overall semantic processing of pages about the topic.

Of course, there are other players on the natural language processing bandwagon, with Powerset one of the most hyped of the bunch. Will Hakia’s approach of providing a unique user experience set them apart? Well, the Buffy page is kind of cool. Whether or not searchers can get comfortable with a new search experience and different way of querying remains to be seen.

Mahalo, the expected people-powered search engine backed by Jason Calacanis, has now gone live in an early "Alpha" test release. In Mahalo, human editors have crafted the top search results for popular queries.

For example, search for [paris hotels], and human editors at Mahalo have assembled a page that lists actual hotels in Paris rather than hotel aggregation/booking sites that you see at Google.

I ran a few queries when talking with Jason about the service last week. Often, the results were impressive. In some other cases, the humans had gone into overkill, listing so many related categories of information that I felt like I was using Yahoo back in 1999.

History: Humans & Search

Human-crafted search results aren’t a new idea, of course. This is precisely how Ask Jeeves used to work. In fact, it was this human element that was crucial to the early success that Ask had back when it initially became popular as an up-and-coming search engine in 1998, alongside Google. Ask’s editors would — as Mahalo editors do now — look at the most popular searches and create search results that editors hoped would be answer the information need.

Ask’s problem was scaling. Having so many editors cost money. In contrast, Google’s link-based automated approach provided good relevancy for both popular and unusual (or long-tail) queries.

Over time, the machine has reigned supreme when it comes to the major search engines. Yahoo’s human-powered directory has been buried in various ways over the years, while Microsoft once heavy-reliance on human editing of top results was long-abandoned in the technological chase after Google.

Scaling Humans

Cost-wise, Calacanis is optimistic he has things covered.

"We have 40 people working on it in Santa Monica and will have 100 by the end of the year. We think we can control the costs," he said. "We can actually make enough money to keep this going."

In particular, Jason talked about his experience in keeping 300 bloggers going though the Weblogs network he used to run as useful.

"It’s not hard for me to keep them [so many editors] focused on a goal." As for funding, if the Google AdSense units currently on the site don’t cover costs, Calacanis says investors ranging from News Corp. to AOL’s Ted Leonsis have given him enough money to run the company for at least five years.

In terms of searches targeted, Jason said the focus remains firmly on the most popular queries that are performed by many people, rather than trying to have a human-crafted answer for everything.

"The goal of the site is not to be a comprehensive search engine. We’re very comprehensive for the most popular search terms. After that, we give the others over to Google or Yahoo," Calacanis said.

Alternative, But Not Replacement, To Google

Give over to Google or Yahoo? Two explanations here. First, Jason doesn’t expect that Mahalo will be used instead of a major search engine like Google or Yahoo. Instead, he hopes it will become a tool people selectively tap into when they want answers for a common, popular subject. He expects those looking to do hard-core research or a hunt after unusual information to turn to a traditional search engine.

"I don’t think it’s possible for people to abandon them," he said, giving the example of someone looking for information about a specific cell phone battery as being too specific for Mahalo to cover.

"It’s not the goal to wipe out Google or be the first choice. Google is the new ocean. You are much better to work with them. We don’t see them as competitive. We’ve got 4,000 search terms, we’re not going to replace them, we know that. But if you compare any of these terms, it’s better to start with us," he said, his "Google as ocean" comment echoing Topix’s Rich Skrenta’s "Google is the environment" observation from earlier this year.

Of course, some people will search for things at Mahalo without knowing it has no answers of its own. In that case, Mahalo will provide Google-powered results, so as not to disappoint. Google’s not a formal partner in doing this, by the way. Instead, Mahalo is simply using the AdSense For Search service that any publisher can tap into. And no — Mahalo doesn’t plan its own ad system, Calacanis said.

Gaining Acceptance

While the goal might not be to replace Google, Calacanis clearly wants his service to get used. How does he plan to grow share? There’s no spending millions on ads similar to what Microsoft and Ask.com have done. Instead, he’s looking to follow the Google model, word-of-mouth.

Plenty of search start-ups have assumed word-of-mouth about their hot new idea or twist on search would be enough to make their companies thrive then been disappointed as they disappeared into obscurity. Why does Calacanis think Mahalo will be different from these others?

"I don’t think they delivered a good product. Compare any of our [human-crafted] search results to Google search results head-to-head. We will be five-to-ten times better," he explained.

What if Maholo somehow beat all the odds and seriously threatened Google? Doesn’t that potentially weaken Mahalo, which is depending on Google to do the hard work of crawling the web and providing relevant results for all those tail terms that Mahalo won’t target?

Calacanis sees this as unlikely — but that’s also where Search Wikia – project backed by Wikipedia’s Jimmy Wales — might come in. Wales is focused more on building an open-source crawling of the web that anyone could use (see Q&A With Jimmy Wales On Search Wikia for more on this). For that reason, Calacanis doesn’t necessarily see himself as "beating" Wales to the punch with a new human-powered service and in fact sees the two projects as perhaps complimentary.

"If his open source results are good and unique and better than Google’s, we’ll use them. We’re defaulting to Google for long tail [query results] because they are the best search out there. I hope that he comes up with something great, because if it is open source, we’d have a great solution," Calacanis said.

Crafting Queries; Avoiding The Destination Trap

Mahalo isn’t just relying on human editors. There’s the ability for users of the site to submit content that should be included on a page, if the editors have overlooked it. But unlike human-powered Wikipedia, these suggestions don’t go live automatically. An editor has to agree to a change and implement it.

"We basically spend four to eight hours on a search term, and that gets us to what I’d consider 60 to 70 percent compete [information for that term]. We’ll rely on the audience to do the rest, though we won’t let them edit the page," Calacanis said.

Pages will link not just to web sites but also to video content, news stories (right now from Google News) and other information. It’s impressive, especially some of the categorical groupings the humans do, though Hakia does some pretty similar and impressive work using technology rather than humans (try a search for iphone, for example — and I hope to finally make time to do my planned write-up on Hakia later in June).

Some Mahalo pages, such as for iPhone, will have "Fast Facts" sections to provide direct answers — though Calacanis said this type of information will purposely be kept to a minimum, so as to avoid the problem that About.com fell into.

In particular, About.com originally launched as the Mining Co. back in 1997, billed as a human-powered alternative to finding stuff on the web — human guides would "mine" the best information out there, hence the site’s original name.

I never viewed the Mining Co. as a search engine but rather a destination site. Calacanis had the same view, when I asked why he thought Mahalo would be a successful search engine when the Mining Co. — About.com — failed to find success in that particular role.

"They strayed from being a guide to the web to being a landing page," Calacanis explained. "We are not going to do that….one of the rules we have is ‘Don’t compete with the destination’," he said. Internally, Mahalo heavily debated having Fast Facts but decided including three or four would be helpful.

Overall, Mahalo aims to cover 25,000 top search terms. About 4,000 have been created already, and the goal is to do 500 per week scaling up to 1,000 per week. This includes revisiting and updating existing terms

Humans Versus Machines

That revisiting is important. I recently talked with Tim Mayer, vice president of product management at Yahoo, about the idea of Yahoo doing more hand-crafting of results. It seems like a no-brainer idea, but Mayer reminded that what’s relevant for a query can often change over time. Google’s Udi Manber, vice president of engineering, made similar remarks when I spoke with him about human-crafted results when I was visiting at Google yesterday.

One example he pointed out was how Google’s human quality reviewers — people that Google pays to provide a human double-check on the quality of its results, so they can then better tune the search algorithm — started to downgrade results for [cars] when information about the movie Cars started turning up. The algorithm had picked up that the movie was important to that term before some of the human reviewers were aware of it.

Overall, the best solution probably isn’t all human or all machine but some combination of the two.

"Humans Are Better" was literally the motto of the Open Directory Project when it launched back in 1998, but that human model hasn’t scaled well. That, along with other human-powered failures, are enough to make Mahalo seem interesting but ultimately not likely to succeed.

Then again, Wikipedia stands so far at a shining example of how humans can indeed come together and produce a quality resource. It’s not perfect, obviously, but it has lots of great information. As I said about Wales’ project at the end of December, so I think about Mahalo — it’s good to see humans getting more of a role in search, and perhaps both projects will ultimately find a way to better blend the best of both words, human and machine.