Jimmy Wales, founder of Wikia, told News.com “It was going to take at least an another year to two before it’s usable by the public, and we can’t afford that right now.” Due to the economy and the need to get to “profitability” right now, Wikia Search needs to shut down. Two full-time employees will be out of a job due to the decision. Wales said that Wikia Search only had 10,000 unique users a month over the past six months.

Is this the end of Wikia Search? Wales said no, he said “return to this again when the economy is good.”

Postscript: Here is the blog post from Jimmy Wales on the news.

WISEApps allow publishers or searchers to enhance search results, in a similar way to Yahoo’s Search Monkey or Google’s SiteLinks enhancements.

For example, the Washington Post WISEApp provides people searching for specific information, such as “Barack Obama” or “John McCain”, with relevant news stories from washingtonpost.com directly within the Wikia Search results page.

Other companies introducing WISEApp’s today are:

• AccuWeather.com: provides people searching for local weather information with full, graphically-enhanced, three-day weather forecasts directly within search results
• AcronymFinder: provides acronyms, abbreviations and meanings for relevant queries
• Amie Street: provides information on independent music for relevant queries
• Creative Commons: provides Creative Commons-licensed images for relevant queries
• Digg: provides recent front page Digg stories for relevant queries
• Indeed: provides job listings for relevant queries
• Kayak.com: provides people searching for information to complete a travel itinerary with the ability to set basic parameters for their trip directly from Wikia Search results pages
• Last.fm: provides band, artist, and song information for relevant queries
• Polar Rose: provides digital photo results for relevant queries
• Thomson Reuters: provides people with financial news for various companies
• Twitter: provides relevant Tweets for various queries
• PleaseDress.me: provides details on tee-shirts for relevant queries
• Snooth: gives people searching for wine direct access to Snooth ratings, pricing and related information directly within Wikia Search results
• Yelp: gives people searching for information on a wide range of local businesses—including restaurants, bars, hotels and more—Yelp ratings and reviews directly within Wikia Search results

More information about WISEApps can be found on the Wikia Search site.

In the notes, it suggests that Google is taking a page from its community playbook: “Already, we’re the cutting edge when it comes to incorporating user feedback into our search results, so much so that Google is experimenting with eerily similar features.”

The new Wikia toolbar allows users to add URLs/pages to the Wikia index (from wherever they are online) and edit or annotate the descriptions and titles of those pages. Users are asked to associate keywords and URLs:

While perhaps simplifying the process of adding to the Wikia index, the toolbar is primarily designed to attract participation and input from users who might be interested in or sympathetic to the Wikia project but use Google or Yahoo as their main search engines.

The Search Evolution So Far

Before going ahead, let me summarize what I covered in my past article, in terms of how search engines have changed over time to create and rank the results you get when doing a search:

• Search 1.0 (1996): Pages ranked using “on-the-page” criteria
• Search 2.0 (1998): Pages ranked using “off-the-page” criteria
• Search 3.0 (2007): Vertical search results blended into regular search results

The evolution above is not perfect. For one thing, some “Search 3.0″ blending started to happen years before 2007. It’s just that in 2007, I felt all the major search engines made the leap into Search 3.0 in a significant way.

As for Search 2.0, looking at off-the-page criteria such as links, Google kickstarted that heavily in 1998. However, some link analysis happened before then, and all the major search engines probably didn’t get on board to using it more fully until 1999-2001. But the launch of Google in 1998 remains the benchmark year in my mind, for that particular change.

The evolution is also only applicable to crawler-based search engines, those that use automation to gather web pages, store copies of them and search through the compiled index to create listings for searches. Yahoo was a major player using human power before 1996 and continued this way for years. Indeed in 1999, a majority of major search engines were presenting human-powered results. This quickly changed as Google grew. Yahoo made its human results “secondary” to crawler-based ones (then provided by Google) in October 2002. Today, all the major US-based search engines depend on crawler-based results.

To cap off the caveats, the evolution above is not the only way search engines can evolve. That’s just how things have largely gone with US-based search engines, which in turn tend to also be the major search engines for most countries around the world. There are exceptions. For example, Naver is the dominant search engine in Korea — and there, listings are largely human generated.

Search 4.0: The Human Factor

Onward to Search 4.0! As I said in my opening, to me this is the move for search engines to make use of human data as part of their ranking systems. In particular, it means human data generated by you, by those you know or by human editors.

Search engines already make use of some human data. All the major search engines, for example, monitor what we click on within the search results. This helps them determine if a particular listing is drawing more or less clicks than would be expected for the position it holds. For example, if the number two listing for a particular query is getting less clicks than “normal” for a listing in that spot, perhaps it’s a bad quality listing that should be replaced with another.

Another example: all the major search engines make heavy use of link data — and that link data is largely human data, humans both “voting” with their links and “tagging” pages by the words they use in the links. Google Now Reporting Anchor Text Phrases and Google Kills Bush’s Miserable Failure Search & Other Google Bombs provide more about how links are used in this fashion.

When I talk about putting human data into search results as part of Search 4.0, I mean things that are more aggressive or active than what I’ve covered above. I’ll start off with the most refined Search 4.0 implementation out there, Google’s personalized results.

Google: Search 4.0 Gets Personal

With Google Personalized Search, the web pages you visit, bookmark and things you click on within search results at Google are used to custom-tailor search results for you. The personalization is not as dramatic as with a place like Amazon, where if you purchase a book once, Amazon seems to continually push similar books like that at you forever. Shifts are far more subtle, mainly to help elevate results from sites you frequently visit.

To understand more, these articles go into depth about the process:

• Google Ramps Up Personalized Search
• Google Search History Expands, Becomes Web History

I’m fairly bullish on personalized search as an important addition to other factors (Search 1.0-3.0) in improving results. For one thing — better or worse — people often judge the relevancy of search results based on ego searches. Does a search engine find your home page, blog and related material when you search for yourself? Does it find your company? Personalized search is an ego search reinforcer. Because you go to your own places on the web often, Google senses that you want them to show up higher in search results, and they do. It’s a genius way to ensure anyone reviewing the service comes away pleased!

Of course, fulfilling ego searches can also be an relevancy advancement, not just a marketing ploy. There’s an excellent chance you’d have better searches if sites you visit more often get a bump in the search results. Personalized search can do this. In addition, over time, personalized search can potentially figure out other sites that are similar to those you visit and give them a relevancy boost.

Since Google expanded personalized search last year, there’s been one further major development. Personalized search uses searches over time to refine results. However, Google also has a system it is testing to refine results based on the last query you did, even if you aren’t taking part in the personalized search program.

“Previous Query” Refinement Coming To Hit Google Results explains more about how this works. It’s been used to improve the ads shown on Google for almost a year now, and it’s currently being tested to refine regular results. Google said that previous query refinement has been one of the strongest signals on how to personalized results so far.

Social Search: Promise Or Hype?

Last year, blogger Robert Scoble kicked off a round of “Facebook’s gonna kill Google” with a series of videos suggesting that because Facebook knows who your friends are, they’ll be able to apply that “social graph” data to improving search results.

The Promise & Reality Of Mixing The Social Graph With Search Engines was my response, a bucket of cold water explaining that using social data wasn’t some new idea that had never been tried before. The article went into depth explaining how Eurekster and Yahoo both assumed search could be “socialized” similar to photo sharing or bookmarking, only to find that wasn’t the case.

Yahoo had little take-up of its social search product. I’ve never seen the company explain why. My own suspicion is that take-up was low because search is NOT a social activity. I believe people tend to search when they have an immediate desire that needs fulfilling, and taking time away from the search activity to “share” with others is a distraction. Consider the person who has a broken water pipe. They might search quickly to find a plumber. They aren’t likely thinking at that moment that they want to tag and classify the search they conducted, much less the plumber they called. They just want the pipe fixed!

Eurekster has said that it found social activity worked better when people organized to build what it calls “Swickis,” search engines that hit only a custom collection of web sites related to a particular topic. Earlier this year, Eurekster formally came out of beta. However, the service has been entirely off-line for almost a week now. Practically no one has noticed, which speaks volumes to its usage and that aspect of the social search potential. Twitter, which some still view as a niche service, can hiccup for an hour and produce reams of blog attention. Eurekster goes silent, and the web stays silent about it.

I’m A Facebookholic & I Have 5,000 Friends

Still, couldn’t Facebook have more luck? For the record, when I spoke with Facebook director of engineering Aditya Agarwal about social search ideas last December, he was far more realistic than outsiders who hype what Facebook could do. In particular, he wasn’t certain how useful the social data actually would be for refining web search.

I plan to do a future article with Agarwal to explore this more. As a reminder, Facebook right now has no web search feature at all. And while it does have an ad deal with Microsoft, our previous Microsoft’s Facebook Ad Deal Doesn’t Include Search article covers how a search partner hasn’t been selected.

Let’s assume that Facebook does select a search partner, which it will need, since trying to index billions of pages and serve millions of queries each day is not an easy task (just ask Microsoft what it’s like to build that from scratch). What could it do with social data?

For one thing, it could monitor what people are clicking on in a potentially more “trusted” environment. Anyone can use web search anonymously, even sending in clickbots to make it seem like some particular listing is super hot. Having to register to be in Facebook and search from within there might make the clickstream data less noisy. But then again, it’s still a fairly open door that someone can walk through, if they want.

Facebook could tailor results based on what friends are searching on. If it knows what you and your 25 friends all seem to select from results, it could ensure those sites get ranking boosts for future searches. That’s very similar to personalized search, except it sounds full of extra friend-goodness, right?

The flaw here is plenty of people have friends on Facebook they don’t know. Some people collect friends for fun (and profit). Some people get friended by others just looking to build up their profiles. Some people you might friend not because you like them but because it’s easier to friend them than say no. Any of these instances can cause “pollution” of the social data that supposedly was going to improve your search results.

Consider also the case of someone who might work at some very conservative company but outside of work is a freeliving, devil-take-all person. Do they want coworkers who are friends to flavor their search results or those friends they hang out with when work is over?

Finally, privacy is an overlooked issue when it comes to social search. People often search for intensely private, personal things using search engines. Search engines are almost like confessionals, where people seek solutions to problems they might not tell real people that are close to them. With social search, do they have to remember to turn off a sharing feature that might be activated by default? And if it’s not on by default, will it get any take-up at all?

In the end, I think there is some potential to tapping into a social network and applying it to search. However, I still remains uncertain how that will unfold. It especially remains uncertain that this is somehow the secret sauce for anyone to jump past the current state of search.

Return To Humans: Hello Mahalo!

Earlier, I’d mentioned how Yahoo started off using human beings to create its search listings in the days before Google existed. Over time, the human soul in search was lost to reliance on the supposed scalability of machines. Anyone who wants to see how much we’ve handed over to machines need only search for buy cialis online on Google. At the moment, the results are littered with online discussion forums that have been “borrowed” by affiliates and others hawking deals.

Those pages will sit there for a day or two or three or potentially weeks, as Google usually tries to find an algorithmic solution to getting rid of them. The idea is you might have to suffer a bit in the short term until a long-term cure is found. But then like a virus that mutates, something else gets through, requiring a new long-term cure.

Enter humans. A human editor, reviewing results like that, can immediately spot junk that should get yanked. Even better, a human editor could act as a curator. How hard can it be to find 10 quality sites that should come up for that or other terms?

That exact human solution, of course, is what Mahalo has been banking on. Mahalo, launched last year, uses human editors to hand-pick top results. For background on the service, check out these past articles:

• Mahalo Launches With Human-Crafted Search Results
• Mahalo Greenhouse: Get Paid For Writing Search Results
• Mahalo Follow: Toolbar Gives You Human-Powered Alternatives To Searching, Surfing
• Mahalo Adds The Social Graph To Search
• Mahalo Adds More Social Features

As part of a talk I do on Search 3.0 and Search 4.0, I have some screenshots from last year that illustrate well how a human can indeed do better than the machines, for some queries. Remember the fires in Southern California at the end of last year. After a series of wide ranging ones, Malibu was hit with a second one a month later. Here’s what those searching on Google got in response:

The news box at the top is great, but sometimes searchers skip past things like this and go to the first “real” result. That’s a story about the Malibu fire early in 2007, not at the end of the year. Other results were largely about the fire of October 2007, rather than November 2007 (which is what many searchers at the time I snapped this would have been interested in).

Here’s Yahoo:

Again, news results at the top, then unlike Google, places you’d expect to find news about the fire — the local paper; ironically a map of the fires on Google Maps that Google itself didn’t return; the Malibu city web site, as well as the fire department.

Here’s Microsoft Live Search:

Very similar to Yahoo — a news box, the fire department, the Red Cross. What’s not to like? Well, let’s look at Mahalo:

Note at the top that Mahalo’s human editors understand there’s a different fire that happened in the past, in October 2007, and offer a link to a page about that. Then there’s a nice list of news sources, followed by coverage by date. Over to the side, a synopsis of the current situation. If you could see more of the page, there was lots of other categorized information.

It’s nicely done. It’s very helpful. And it was created with a human thinking about what other humans might want to see, rather than machines guessing.

Scaling Humans

So is Mahalo founder Jason Calacanis onto the Google-killer, human crafted results? No. I think human review can be part of the solution, part of the Search 4.0 addition to what we have out there already — but humans can’t craft pages for every possible search. In addition, it’s hard to keep those pages maintained once they’ve been made. It’s also easy to cross over from being a search resource that points to other resources to becoming instead a destination site. I think a good search engine avoids that (and Who’s Ranking For Knol? Hello, Wikipedia! has more on this topic).

Mahalo can also be overwhelming. Try a search for hillary clinton and there’s category after category. Background links. News links. Photos. Videos. Bio links. Blogs and message boards. Plus, there’s even more. I think at some point, you want your search engine to make some key choices for you, not flood you with so many that you don’t know where to begin.

Another issue is that what Mahalo’s human editors do, machines can get close to. Hakia especially stands out here. Search for hillary clinton there, and you’ll see how listings are grouped into categories like Awards and Biography without humans being involved (and see Social Networking Through Search: Hakia Helps You Meet Others for background on how Hakia works).

More Humans

There is another major search project involving humans: Search Wikia. Backed by Wikipedia cofounder Jimmy Wales, the service aims to involve humans in rating pages, annotating them and helping determine the ranking algorithm for choices the machine side of the project makes.

The articles below have more background on the service:

• Search Wikia: Not Even A Remote Threat To Google
• Search Wikia Adds Alpha 0.2 Features & More

Right now, the quality of the service is poor, as Search Wikia itself readily admits. There’s still lots of work to be done — and even with it, it might never succeed. But allowing humans into the process is, in my view, a good thing.

Indeed, even Google understands this. Last year, Google started doing some education about how human “signals” are already incorporated into its algorithm (see Google’s Human Touch and Google & Human Quality Reviews: Old News Returns). Aside from this, last year it also started testing a way for people to annotate search results — add those they like, remove some, suggest other ones.

Watch Personalized Search

Overall, there’s a role for humans, a way for them to be in the search process to enhance results. Actually, there will be several ways for them to be involved. Exactly how remains to be seen, of course.

Of the things I’ve outlined — personalized search, social search, human editors — I think personalized search is the one that will emerge as the major part of Search 4.0. That’s not to discount other things being tried, and they’ll contribute in some ways. But to me, personalized search has the most potential for another big relevancy leap. We’ll see!

Below you can hear Jason’s presentation yourself, then you can hear the Q&A portion that covered search marketing and human powered search. Note that the video production could be better. Hey, I just got a Macbook — it’s my first time playing with what it can supposedly do. Do don’t hassle me over the titles that could be better. Also, I will get the entire session up with the presentations from Jimmy and Steven, along with the further Q&A. But first, Jason:

The Q&A:

And here’s some of the reaction to initial reports of his comments, before anyone not at the show could hear what he said in full:

• My Plea to SE Conferences: Turn Calacanis Away, Eric Lander
• My Plea to SE Conferences: Turn Calacanis Away, Sphinn discussion on post above
• Why Calacanis Should Not Be Allowed to Speak at Conferences, Slightly Shady SEO
• Why Calacanis Should Not Be Allowed to Speak at Conferences, Sphinn discussion on post above
• Jason Calacanis has pissed off SEO’s… again. Seriously., Sphinn
• Srsly are SEOs Really That Easy To Manipulate?, ShoeMoney

And here’s further coverage of the show (for initial coverage from the first day, see SMX Social: Mahalo To Do Microformats, Search Wikia Adds Alpha 0.2 Features & More):

• 20 Take-Aways from SMX Social Media – Scott Clark, Scott Clark
• Evangelist – The Marketer’s Role in SMM, BruceClay.com
• Micro Communities, BruceClay.com
• Wikipedia, Yahoo Answers & Answer Sharing, BruceClay.com
• Social Media Expo (SMX) Report from Long Beach, California, Wpromoter
• SMX Social Media Conference in Long Beach, MindCradle
• Rand Fishkin | Jason Calacanis | SMX Social, Gary Pool
• SMX Social Media Coverage Round Up, BruceClay.com
• SMX Social Media Pics, Scott Clark @ Flickr

Jason Calacanis of Mahalo, Jimmy Wales of Wikia Search, and Steven Marder of Eurekster all spoke at the end of the day on our Social Search: The Human Challengers panel. Lisa Barone provides live blogging coverage in her Keynote – Social Search: The Human Challengers post at the Bruce Clay blog.

During that session, Jason shared that Mahalo is providing microformat support. Mahalo Adds Microformats from Mahalo’s Sean Percival covers how this works, complete with screenshots. Check it out! See also further discussion on Techmeme.

Also during the session, Jimmy Wales shared that Wikia Search has added several new features as part of a new alpha release, one he dubbed "Alpha 0.2." Wikia Search doesn’t seem to have news up yet on its own site, but Wikia Search Launches Major Enhancements to Search Alpha from CenterNetworks covers how you can do things like preview pages and copy sections of them into search results. You can also add related search terms to queries, add pages you think should be added to a search result, and delete material you think doesn’t belong.

Scary, freaky? You bet — but Jimmy also covered how, like Wikipedia, the history of all this can be seen, which potentially provides some self-policing. Try the new features on the test site here. And further discussion can be found on Techmeme.

For other session coverage from the conference, see:

• Social Media Marketing Essentials, BruceClay.com
• Linkbait – Chumming for Traffic on Social Media Sites, BruceClay.com
• Extra! Extra! The Social News Sites, BruceClay.com
• A Marketer’s Guide to Social Bookmarking, BruceClay.com

We’ll also do a Day 2 post tomorrow with further coverage and add to it anything we’ve missed. Also, you can keep up with live blogging both at the Bruce Clay blog and via Twitter. Yep, Twitter. Michael Gray is diligently microblogging on our SMX twitter account.

For now, though, Search Wikia is essentially useless as a search engine. What makes it interesting are some of the ideas and approaches that the service is exploring. I’ll not bother bashing the search results that even Search Wikia itself admits are poor. Instead, I’ll take a look at some of the problems and issues with the current state of web search today that Search Wikia is attempting to address.

“We’re not producing a Google killing search engine”

That quote above is from Search Wikia founder Jimmy Wales himself in a Q&A with Jimmy Wales a little over a year ago. In the Q&A, Danny probed Wales for his motivations for creating a new search engine. Wales singled out several issues, which have since evolved into “organizing principles” for Search Wikia (more on those below).

One problem, according to Wales, is that search engines are effectively black boxes—you punch in a few keywords, the search engine does its thing, and serves up results without any explanation. The rub is that you have no idea what just happened, and sometimes search fails miserably.

“For certain types of searches, search engines are very good,” said Wales. “But I still see major failures, where they aren’t delivering useful results. I think at a deeper almost political level, I think it’s important that we as a global society have some transparency in search. What are the algorithms involved? What are the reasons why one site comes up over another one?”

In a New York Times article today, Wales said, “I think it is unhealthy for the citizens of the world that so much of our information is controlled by such a small number of players, behind closed doors… We really have no ability to understand and influence that process.”

Which leads to the next problem: Lack of community involvement. This year, we’ve seen an explosion in the popularity of social media sites, including efforts by the major search engines to include people in the mix, in many ways. But not to the extent that Wales feels is appropriate.

How has Search Wikia addressed these problems? Let’s take a closer look at Search Wikia’s “four organizing principles” featured prominently on the home page of the Search Wikia Labs Wiki.

Four Organizing Principles (TCQP) – the future of Internet Search must be based on:

• 1. Transparency – Openness in how the systems and algorithms operate, both in the form of open source licenses and open content + APIs.
• 2. Community – Everyone is able to contribute in some way (as individuals or entire organizations), strong social and community focus.
• 3. Quality – Significantly improve the relevancy and accuracy of search results and the searching experience.
• 4. Privacy – Must be protected, do not store or transmit any identifying data.

Let’s look at each of these four principles and how Search Wikia is dealing with them in more detail.

Transparency: Fostering democracy, or handing the fox keys to the henhouse?

Search Wikia is an “open source” search engine, meaning its users are free to modify and adapt the software, search results, and other content compiled by the search engine.

In the software world, the open source movement has aroused near religious fervor in some. The idea that software source code should be freely available for anyone to view, modify, adapt, and share it with others has merit, as evidenced by the success of the Linux operating system. But open source hasn’t really displaced so-called “closed-source” software in any meaningful way. Microsoft is still by far the dominant operating system provider, despite major efforts by the open source Linux community.

And while open source efforts may work well for something like an operating system, where the software must execute clearly defined functions with outcomes that are pretty much universally agreed-upon, search results, by nature, are highly subjective, with few standard techniques or methodologies apart from basic matching, sorting, and other relatively straightforward functions that are just the building blocks of a search engine. And in the end, one person’s perfect result may be complete garbage for someone else.

Further, search engines are such complex “entities” that I’d argue there is no single individual within Google, Yahoo, or any other major search engine that actually understands everything about how the search engine works. Computer code is highly technical, and while a programmer can often get a sense of what code does by reading it, code must be run and tested along with all of the other interrelated elements to truly see how it works. I have serious doubts that a volunteer corps could make “improvements” to search engine code without introducing unintended consequences, or worse, intentional biases that may seem perfectly reasonable to the volunteer coder yet totally unacceptable to the programming community—let alone searchers who don’t participate in the process.

It’s somewhat easier with text: The Wikipedia community does a good job of monitoring and fixing incorrect entries in its encyclopedia, but how could a programming community do a similar job of policing given the highly subjective and extremely technical nature of coding for a search engine?

And as searchers, do we really want or need that transparency? Ten years ago I could look under the hood of my car and fiddle with my engine when I wanted to modify something. Today, just about every system in my car is computerized, completely inaccessible to my tinkering. But given the virtually maintenance-free operation of my car, I’m perfectly happy with that change and don’t long for the lost days of “engine transparency” at all.

In fact, though I’ve studied computer science at the graduate level, I have a difficult time deciphering the apparent “relevance score” that appears next to each Search Wikia result. Click on this link and you’ll see a page that shows you how the open source Nutch search engine used by Search Wikia went about calculating the relevance of a particular result. “Transparent,” yes, but also totally opaque to anyone other than a programmer who’s familiar with Nutch.

And what about spammers? I know several search marketers who lean toward the dark side who would absolutely love to see exactly how search algorithms work. I’d be willing to bet a sizable sum that they’d be able to figure out a way to game the system to their advantage in less than 24 hours.

In fact, it has been less than 24 hours since the launch of Search Wikia, and spammers have already discovered this new resource: Check out the Recent wiki changes page to see what various enterprising “community members” are adding to Search Wikia.

Community: But we already have community!

All of the major search engines save Ask.com have launched numerous “community” features over the past several years. Yahoo is by far the most aggressive in rolling out community-influenced services, such as Answers, My Web, and others. Google and Microsoft have their own initiatives.

Further, most have offered technically-savvy users the ability to both use and create things like widgets, APIs, and other extensions of the search engines’ functionality. These initiatives offer both community, and, to a certain degree, transparency, for anyone with the skills and interest to avail themselves. Granted, none are playing the community gambit to the extent that Search Wikia is, but it’s not clear that community in and of itself is a beneficial thing for search services. Virtually all “social search” services introduced since the beginning of the web have either failed or not garnered very much traction. This doesn’t mean that community isn’t useful—just that it isn’t going to be a silver bullet for improving the quality of search or attracting users to Search Wikia.

You can participate in the (non-programmer) Search Wikia community in many ways. You can post bug reports, suggest sites for a “whitelist” of sites considered “must have” authorities, or create a “mini page” (essentially a Wikipedia-like entry summarizing a topic).

Search Wikia is also a social network of sorts—when you register, you are given a profile page that lets you add information about yourself, which then in turn becomes part of search results if people search on your name.

When you register for a Search Wikia account, you can register both a user name and your real name. Somewhat bizarrely, though, you see this message next to the real name field: “Real name is optional and, if you choose to provide it, will be used for giving you attribution for your work. This field must be filled in to use Search Wikia.” So, your real name is optional but you must fill it in if you want to use the service. Right.

A huge problem with community-driven sites is politicization: A certain group of users invariably comes to dominate discussion and exerts undue influence over less “powerful” users. We have seen this for years in the Open Directory project; more recently in Digg and even Wikipedia itself. What starts as a noble experiment in participation typically degrades into a species of the tragedy of the commons.

Quality: Is it tuh-may-toe or toh-mah-toh?

To me, improving quality is Search Wikia’s most laudable goal—and one that it is far, far away from achieving. Sure, search quality at the major search engines is OK these days, but we’re still a long way from being perfect, and anyone you talk to from the major engines who’s honest admits this. But quality is such a vague concept, with relevant results so much of an “eye of the beholder” issue that, while notable, “improving quality” is also virtually a meaningless, almost gratuitous goal. Turn this principle around: Does anyone set out to build a search engine with poor quality? Quality is the price of admission these days, not a differentiating feature. If you don’t have good quality from the outset, you’re dead—and it may have been a mistake for Search Wikia to open itself up at this early stage when its search results are just awful.

Search Wikia has an advantage in that the thousands of Wikipedia users may opt to chip in to help improve the quality and fill in the missing blanks in Search Wikia. But this will need to happen quickly for Search Wikia to gain traction. Wikipedia survived and ultimately thrived because it filled a need (most online encyclopedias were fee-based until Wikipedia). Unlike Wikipedia, Search Wikia isn’t filling any unmet need at this point, and it’s just not clear that volunteering will ultimately reward users.

Privacy: Don’t ask, don’t tell

Search privacy has become a hot issue once again over the past year, with most of the major engines announcing new, stronger policies regarding the data they capture and store. Search Wikia says it is going farther than most of the majors to protect privacy, but there are some blazing red flags with this initial version.

Although the fourth principle states that privacy “must be protected, do not store or transmit any identifying data,” Search Wikia has no privacy policy statement—at least one that I could find. If you log in, you will see a “privacy” tab, and clicking this takes you to a page that gives you lots of apparent control over your privacy settings. I say “apparent” because almost all of the settings are related to who gets to view your personal information that you enter in your profile—there’s nothing related to who gets to see your search queries, browsing history, and so on.

Also disturbing is the broken “Terms of Service” link that appears at the bottom of each page, which currently simply just takes you to the top of whatever page you’re viewing.

In making privacy a core principle, I would have expected a lot more disclosure, transparency, and reassurance. Instead, you have to take Search Wikia’s commitment to privacy pretty much completely on faith at this point.

The Bottom Line

If you’re looking for a worthy alternative search engine, don’t bother with Search Wikia at this point—its results will make you long for AltaVista circa 1994.

However, if you support the principles that Search Wikia is pursuing and you feel like volunteering for a pretty ambitious experiment, definitely register and give Search Wikia a try. Be prepared to spend a fair amount of time learning how to use the system—and in dealing with other users who may not agree with you or don’t like your work.

If Search Wikia ultimately does gain traction and can live up to its guiding principles, we may yet see the emergence of a truly useful alternative search service. Just don’t hold your breath for that to happen any time soon.

For now, you can get more coverage at Techmeme, but Chris’s review will be live soon here at Search Engine Land.

Postscript: Chris Sherman’s piece is up, where Chris concludes Search Wikia: Not Even A Remote Threat To Google.

Blekko

What we know so far about Blekko isn’t much, and TechCrunch has the most details in its The Next Google Search Challenger: Blekko post from yesterday. Apparently Rich founded the company in September 2006, along with five other former Topix employees, after he left Topix in June.

Rich told TechCrunch not to likely expect anything public until 2009. I agree with Michael Arrington at TechCrunch that Rich has a track record that makes him well worth watching. The Open Directory was an initial success, though the model didn’t scale well. Some of that was within the founders’ control but had more to do with AOL’s lack of backing. The company should be dragged into the International Court Of Search Crimes and be forced to sell the ODP to someone who will support it properly. Topix has built a reputation and is still standing and succeeding — though I’d say it still has far to go to seriously threaten Google or Yahoo.

Rich adds a bit more in his Why Search? post today:

Having just spent 5 years in the media space, I’ve come away with the idea that editorial differentiation is possible. But the editorial voice of a search engine is in the index…so it has to be algorithmic editorial differentiation.

So far, it doesn’t sound like a social networking play like some of the others. We’ll be watching, Rich. Also see discussion today on Techmeme.

Powerset

Powerset is now a classic example of why you WANT to be a stealth start-up and say little. That’s because when you get too much early press — in part through your own doing — then fail to deliver anything, the hype can swing back at you hard.

The company came to light back in October 2006 via VentureBeat, with the twist being that natural language search would be the way forward. That caused me to write a long rant about the hype of natural language search in reaction. From the top of that:

This is a rant. It’s a rant from over 10 years of watching people trot out natural language search as the "killer" solution to the current state of search, something that’s happening once again with Powerset. That’s a search engine you can’t even use at the moment, but the hype will no doubt continue. To counteract that, my thoughts on and some history about natural language search.

Natural language search makes a compelling pitch for those who really don’t know search or haven’t heard the natural language mantra before. I’ve seen the pitch time and time again. You:

• Pick out an example that shows how "bad" search is on an existing search engine
• Demonstrate how natural language search would work better on your service
• Sit back and collect the press attention

I then went on to detail how natural language search had been hyped and tried over the years. The short story is this: It doesn’t take much natural language analysis to figure out what someone wants when they type in "britney spears nude" or "hotmail." In addition, by and large I don’t believe enough people will change their basic search habits to enter long sentences when searching any time soon.

Since that time, we’ve pretty much had nothing out of Powerset other than the launch of Powerset Labs in September 2007. That launch hasn’t produced any cool applications that I’ve seen or heard about, nor much buzz. Instead, in November, we got a management shake-up.

For a more formal chronicle of the company’s developments, check out this area at VentureBeat and these search results at TechCrunch.

Finally, while I’m harsh above on Powerset, I actually had a long visit with the company in the middle of last year and was deeply impressed with the effort going on there. I’m still working on a long write-up to explain what’s happening. But in a nutshell, Powerset is trying to literally comprehend or understand each page on the web.

Today’s search engines don’t know what a page is about by reading words. They’re more or less doing pattern matching — finding pages that contain words similar to what you search for (or pages relevant to those words based on linkage). Powerset literally is trying to read and understand what a page is about the way a human reads a page and knows it is on various subjects.

I don’t see that as making it a better search engine that Google. Instead, I think it may eventually give it the ability to create a unique "auto-Wikipedia" style site, assembling knowledge pages on any subject automatically. I also think that there will eventually be some search benefit in comprehension of pages, but exactly how that will play out I suspect is part of being with an existing search engine and a more traditional model. With the array of patents Powerset has lined up, I suspect it will eventually get acquired by Google, Yahoo, or Microsoft rather than rollout its own product. But we’ll see.

Hakia

Like Powerset, Hakia has played the natural language search game. Unlike Powerset, it has a product anyone can use — live since at least the middle of 2006.

Again, I’ve been working on a long write-up on the inner workings of Hakia and have yet to finish it. It’s complicated, and I mainly want to cover what I find to be the real use of their technology — the ability to create custom "gallery" pages and understand those are related to particular searches.

It’s easier to show you what’s impressive. Search for hillary clinton, and you get a nice page showing news, her official site, biography pages, blogs & fan sites, news & interviews, and more. It’s very Mahalo-like, except it doesn’t require human editors like Mahalo and predates Mahalo by a year.

That categorization is something I know the major search engines could do, if they wanted. So far, they don’t. And so far, despite Hakia talking about its rising traffic, it has yet to make a serious mark. Moreover, in October, it made a serious shift to allow social interaction with its results. That’s a sign that the original plan that "natural language will win all" has failed to do so; therefore, another twist is needed.

Social Networking Through Search: Hakia Helps You Meet Others from Vanessa Fox here at Search Engine Land covers the change, plus it gets into the natural language indexing stuff I mentioned earlier that makes Hakia unique, plus has examples of gallery pages.

Mahalo

Credit to Jason Calacanis. He said he wanted to take on Google, then wasted no time getting Mahalo rolled out. OK, he also says he’s not taking on Google — just focusing on the top searches that he thinks would be better with human review. Sure, you aren’t taking on Google, Jason.

To date, Jason reports that Mahalo’s traffic is growing and strong. But to date, I’ve certainly see no webmasters taking about what a traffic driver Mahalo is. It would be early to call it a raging success, but it’s a nice alternative to have. Indeed, later this month I’ll finally finish my Search 4.0 piece that picks up from the conclusion of my Search 3.0: The Blended & Vertical Search Revolution article last November. I’ll show some examples of how the human element at Mahalo can and has kicked some Google and traditional search engine butt — though also how it isn’t the panacea some expect.

Some of our past coverage of Mahalo:

• Mahalo Launches With Human-Crafted Search Results
• Mahalo Greenhouse: Get Paid For Writing Search Results
• Search Spam Fight - Mahalo: 1; Squidoo: 0
• Mahalo Follow: Toolbar Gives You Human-Powered Alternatives To Searching, Surfing
• The Promise & Reality Of Mixing The Social Graph With Search Engines
• Mahalo Adds The Social Graph To Search

Search Wikia / Wikia Search

Wikipedia founder (as he prefers to be called; Wikipedia itself calls him cofounder) Jimmy Wales made waves a year ago when he said he’d take on "closed" Google with humans and a transparent search engine. Called Search Wikia (but, confusingly, it’s also called Wikia Search), Wales has grabbed attention from the press over the past year. Slamming at Google as a scary closed thing gets you good mileage, especially when you helped establish Wikipedia, a threat Google takes so seriously that it may launch its own Wikipedia-style site, Google Knol.

Now Wikia Search is at hand. A private "pre-alpha" test started in late December, an invite-only thing I still find odd for a service that’s supposedly all about the "transparency." But on Monday, the general public will finally get a look at whatever Wales and his team have concocted. In the meantime, while Wales still hasn’t posted any news since July 27 to the "news" section of Search Wikia, press reports tell us so far:

• Only a tiny 50 to 100 million pages will be indexed at launch. The major search engines today have tens of billions of pages indexed. (AP)
   
• There will be a high degree of human editorial influence, though whether that’s over the algorithm or the search results on a per-query basis remains to be seen (CMP)
   
• An early screenshot suggested that Search Wikia might be evolving more into a Facebook-style service, perhaps with some ways for users to share results (Matthew Buckland & Wired)

Some of our past coverage of Search Wikia:

• Q&A With Jimmy Wales On Search Wikia
• Search Wikia Takes Steps To Crawl; Acquires Grub
• Search Wikia Gets Open Source Categorization Software
• Search Wikia Launches In 2007 With Private Beta

Cuill

Arguably the stealthiest of the stealth start-ups, Cuill (pronounced "cool") has an impressive pedigree with its three founders: Tom Costello of IBM’s WebFountain project and Anna Patterson and Russell Power of Google’s TeraGoogle project, its massive search index. And last year, former AltaVista founder Louis Monier — who later went to eBay as its first eBay Fellow, then to Google — jumped ship from Google to join Cuill.

I talked with Cuill earlier this year to understand a bit more about what they are doing, but the details are still being held very closely. The main difference between Cuill and everyone else I’ve named above is that Cuill is founded by people who understand and have dealt with firsthand the challenge of indexing billions of documents.

Cuill recently took on more funding. Louis is also going to be doing a keynote at our SMX West search marketing conference, held in Santa Clara, California from Feb. 26-28. I’m thrilled to be having him since there are only a handful of people who have worked for the "old" Google (AltaVista), the current Google (when he was at the Big G), and a potential future Google (Cuill).

And The Winner Is…

If you think the future of search is on smart automation, Cuill’s definitely one to watch, and perhaps Blekko as well. If you think it’s the growth of humans, Mahalo and Search Wikia are your better candidates. The reality is that success will likely be a blend of the two. For the human services, a real open source index would be a big help — see Google: As Open As It Wants To Be (i.e., When It’s Convenient) for more about this.

But the reality is that all of these services will have an incredibly tough time to beat Google.

Google came along at a very special time, as I’ve long written. It had better technology at a time when all the search engines had abandoned improving search, since that was seen as a loss leader. The money was in portal features.

Today, search is a multi-billion dollar industry. If someone with a serious search threat comes along, you buy them (such as with YouTube), or you start to develop your own rival if it seems a real threat. Google’s not omnipotent — but you’ve already got a space where it’s Google, Yahoo, Microsoft, and Ask all seriously fighting it out (and the latter three, despite their funding and experience, still struggle against Google as being synonymous as a trusted search brand for most users).

To date, Google is the real exception of "a better mousetrap wins." It’s far more likely the companies above, if they do gain traction, will end up being purchased for a large amount by one of the existing "search utility companies."

Jimmy Wales, cofounder of Wikipedia and driving force behind Search Wikia, said in the email that Search Wikia is “going to be letting people in slowly over the next few days and we are aiming for a January 7th public launch.” Technically, they launched just before 2008, which makes for a 2007 launch. The public launch is expected January 7th.

For more coverage on the Search Wikia launch, see Techmeme. For our past coverage on Wikia Search, see our Search Wikia section and Q&A With Jimmy Wales On Search Wikia, from when the service was announced a year ago.

Postscript From Danny: Interestingly, Search Wikia’s home page has no mention of the beta happening, which is odd for a service that lists “transparency” as its first of four organizing principles. However, it is mentioned in the public mailing list for the service here.