B2B sites tend to be more difficult for visitors to find what they’re looking for. Perhaps it’s because things don’t always fit neatly into more intuitive consumer categories. Perhaps it’s because B2B sites are often laden with so much diverse information. While site owners can engineer enhanced usability, better optimize and structure content, or create better organic landing pages, none of these options is a quick fix. Google’s Site Search offers a quick, inexpensive way to keep visitors engaged and (hopefully) get them quickly to their destination on your site.

We’ve all clicked on promising organic search results and been quickly disappointed that the landing page doesn’t contain what we’re looking for. In many cases I’ll often land at a site I’m fairly certain contains what I want, so I’ll take a few clicks through the site’s navigation. But if I don’t find what I want in a few clicks, I don’t have the patience to keep searching. I’ll go to another site. All of us see these visits in our analytics, too. A four-page, 20-second visit. Then, they’re gone.

Site search functionality offers a way to keep visitors engaged a while longer. If visitors don’t quickly find what they want through navigation, they may try the site’s search tool. Many B2B visitors will go to the site’s search tool right away as an alternative to navigating to find an answer.

While many larger sites have already have site search functions, more often than not I’ve been disappointed with their search results. When I’m looking for a specific product or service, I’ll get hundreds of search results, but the first 30 results will be investor news releases or obscure technical articles. The results aren’t relevant to my quest. Not only do I leave without my desired answer, I’ve also formed some negative perceptions of the company and its website.

Last week, Google relaunched its rebranded custom search engine as Google Site Search. I think it’s a good answer for many B2B sites. It doesn’t cost much. Pricing depends on the number of pages indexed and the number of annual queries. For a site with less than 5,000 pages and less than 250,000 annual search queries, the cost is $100 per year. Pretty reasonable.

Getting all of your content indexed by Google can be a challenge, especially with large B2B sites. Google Site Search offers the opportunity of deeper site indexing for site-specific search. While this deeper site indexing won’t get more pages indexed by Google or help you in your Google rankings for web searches at Google.com, it will help you ensure all of your pages are reflected in the index of your site’s Google Site Search. This means searchers will get different (and likely better) results using Google’s Site Search on your site than if they used Google.com to search for information on your site (e.g., incorporating site:www.yoursite.com into the Google query).

Google’s site search also gives site owners the opportunity to “bias” the search results in a couple ways. For sites in which new content is typically more important, site owners can ensure search results are more heavily weighted to newer site content. Site owners can also bias search results to reflect certain sections of the site more than others, e.g., product-related pages more than company-information pages. This can help drive searchers more quickly to revenue-generating pages.

If you don’t have search capabilities on your B2B site, it makes sense to spend $100 to try it. There’s not much you can do for $100 these days. So try it out. Then watch your analytics. Notice what visitors search for. That alone is great information. Also, see if your bounce rates decline, or if the average time on your site goes up materially. And watch your conversion rates.

If you already have search functionality on your site, you still may want to test out Google Site Search. Set it up and do some comparative searches. See if you think the search results are more relevant or if the user experience is better. I’m not sure if Google’s Site Search will be better than what you already have (and I’m not trying to sell Google’s Site Search), but again, for $100, it’s worth a test.

Google has announced a beefed-up version of Google Custom Search, a site search product aimed at small and medium sized businesses. The new service is hosted by Google and offers all of the same functionality of the free Google Custom Search product, with a few key differences:

Customized search results. You can change the look and feel of search results to match the underlying design of a website, using a relatively straightforward XML API provided by Google.

Optional ads. With the free version of Google Custom Search, Google displays ads alongside search results. With the Business Edition, ads in search results are optional.

Dedicated enterprise support. The new service offers both email & phone support,

Optional Google branding. Don’t want the Google logo on your search results? You can easily turn it off.

Custom Search Business Edition starts at $100 a year for searching up to 5,000 pages and extends to $500 for 50,000 pages. Larger volumes of pages are supported through Google’s enterprise sales group. For smaller sites, businesses can sign up using a credit card for payment, and get a site search up and running quickly using a simple online interface.

To see an example of Google Custom Search Business Edition in action, see Holiday Home Rental, an online directory of 25,000 holiday rental homes with more than 150,000 pages of content. More information about the new service at the Google Custom Search Engine page.

Enterprise search provider FAST Search & Transfer has acquired personalization platform and recommendations engine AgentArts. The technology will be folded into FAST’s various enterprise search offerings, which include a range of site search and monetization options for online publishers. The company also has a mobile search partnership with InfoSpace.

FAST has positioned itself as a complete platform to help publishers of all stripes essentially compete with Google and avoid dependence on AdSense and other third-party contextual ad networks.

According to the Wall Street Journal (subscription required), “Inform Technologies LLC, which is based in New York, said 16 online publications plan to include its new search function. Among them are WashingtonPost.Newsweek Interactive’s Newsweek.com and Conde Nast’s Portfolio.com.”

Inform began life a couple of years ago as a consumer destination site and news aggregator. Failing to gain traction (partly because it was too complex), it more recently reinvented itself as a publisher site search and platform provider.

The idea behind using Inform’s technology is to provide better on-site search and related news/content search from third-party sites. That potentially turns all the Inform-powered sites into news aggregators themselves. The aspiration is to provide a better user experience and more utility and boost page views, which in turn means more ad revenue for publishers.

Corporate spending for electronic-discovery services, estimated at roughly $1.77 billion in 2006, is expected to increase by more than a third to $2.42 billion this year, according to the Wall Street Journal. The reason? New Federal Rules of Civil Procedure passed by Congresss that are accelerating deadlines for companies to identify and describe electronically searchable information that can be used as evidence in a lawsuit.

From the WSJ’s Search Software Gets Boost From New Rules:

“The rules require that all relevant, nonprivileged and reasonably accessible information that can be used as evidence must be produced. In corporate lawsuits, parties are given 99 days to agree on a formal procedure outlining what information is to be produced, when it is to be made available and the form in which it is sent to the other party. These rules apply to both U.S. and foreign companies that are sued under federal law.”

The electronic discovery market is split between software and service providers.

Via PaidContent, news that Reuters is to acquire text analytics company Clearforest, part of creating a new strategic group around search. Leading that group is Gerry Campbell, the former long-time head of AOL’s search efforts:

As part of its drive into this space, it has created a new strategic group within Reuters and appointed former head of search for AOL, Gerry Campbell, who will oversee the integration of ClearForest and drive this innovation.

PaidContent made references to Clearforest having been working with Reuters to "boost search rankings for its business stories." To be clear, I don’t think that Clearforest has been providing external SEO-style ranking support. Instead, the Clearforest tools seem designed for those who want to make content within a web site (or a particular database) more easily found by visitors.

Reuters is also launching a new tool designed to scan for market data. Reuters introduces machine-readable sentiment scan; acquires ClearForest from Finextra covers this more.

Campbell — who was senior VP and general manager of AOL Search — joined Reuters in late 2006, according to his verified ZoomInfo profile. That was a quiet move I don’t recall reading about. He turns up more publicly at a new company coincidentally the same day similar high-ranking Yahoo search vice president has turned up elsewhere. Our post Yahoo Search VP Andrew Braccia Resigns, has more on that.

The relationship between enterprise search company Autonomy and video search company Blinkx has always been confusing. PaidContent.org now reports that the "weirdly secretive relationship" (I agree) is getting clearer. Autonomy is to exercise an option to take over Blinkx, then appears to be spinning some consumer-facing search technology that its owns (and I believe Blinkx was licensing) into an independent company Blinkx, that will go public in London.

The Financial Times goes into more details on the deal, plus Autonomy offers a press release here. It all still seems as clear as mud. A few more details, from what I can figure out, below.

Autonomy doesn’t have any consumer-facing products, as best I can tell. Here, you look and try and find them. In the late 1990s, it pitched "agent" technology that was going to bring back items of interest to searchers. That never took off. Then in 2000, it offered this odd Kenjin tool that would analyze what you were viewing and offer suggestions. That never gained acceptance.

When Blinkx first came out, it was like Kenjin with a new name. As I wrote at the time:

Want to search your hard drive? Blinkx is designed to do this, plus provide the ability to search the web, as well. It’s similar to what many expect Microsoft will one day try to do or that Google itself is rumored to be doing.

I’m wary of tools like Blinkx that are pitched as if they will eliminate keyword searching. Kenjin was exactly such a tool that Autonomy rolled out back in 2000 promising to put "search engines in the dustbin," in one PR quote that I recall. Read the release [NOTE: no longer online] from the time for more hype about how our lives were going to change.

The idea was that you’d show Kenjin a "perfect" document, then have it go out and find others that matched. Kenjin failed, and the reason is pretty clear. How do you show it the perfect document, if you don’t have it to show already? I know — you use a search engine!

Blinkx is apparently using some of Kenjin’s technology, plus resurrecting some of that hype — but this time, a search box is also present. Reviews of the products can be found here. Gary Price has a nice long rundown here,  and there’s a recent Boston Globe review here.

I downloaded a copy but couldn’t get it to work — you might have more luck. The site also offers an interesting online search feature for those with broadband connections, with some fun visualization tools to play with.

Some time after that, Blinkx transformed into a video search tool. Along the way, poor Gary Price tried to figure out the relationship between the two companies, getting this statement from Blinkx CEO Suranga Chandratillake:

Autonomy is not one of blinkx’s shareholders. We [blinkx] enjoy a close relationship with them (Autonomy) but that’s because (I was there for years (including as US CTO) and have lots of friends there, (b) we are an OEM customer of theirs, and so depend on them in a number of ways technologically. Under the terms of the OEM agreements, under certain circumstances, Autonomy does have an option to invest in blinkx.

So now Autonomy seems to want to exercise that option, no doubt looking at the $1.65 billion Google paid for YouTube and wondering if it can tap into some of that money. But what is this consumer stuff beyond Blinkx that the Autonomy press release talks about:

Autonomy’s consumer division is formed from Autonomy’s research and development related to the application of Autonomy’s award-winning IDOL technology to the consumer space, and blinkx.

Here’s background on IDOL. From what I can tell, it goes back to some of the text analysis technology used in the Kenjin product, the original Blinkx tool and no doubt some of the text analysis that Blinkx does as part of video transcription search.

The FT article says that "Blinkx will be given exclusive rights to the technology, everywhere outside China." I think this means exclusive for consumer facing products, since Autonomy still seems to use this technology for its core operations

The FT article also says "Autonomy’s technology does not rely on text and keywords but uses mathematical formulae to detect patterns in any type of information, including pictures and sounds." Actually, I don’t believe this is correct. When it comes to pictures (specifically, video) and sounds, my understanding is that Autonomy is still examining textual content that describes that material (transcripts). It very much relies on texts. But rather than just doing straight keyword matching, it looks beyond the exact words to find meanings.

Autonomy used to strongly pitch that its Bayesian pattern matching analysis could figure out a document was on certain subjects, as you still see here, allowing it to find other documents with the same subject "fingerprint." But ultimately, this was still a word-based search technology.

This was back in the early days of search engines, and the pitch was something Autonomy especially used to distinguish itself and suggest it would outlast the big contemporary search engines around at the time. Yes, Lycos, Infoseek, Excite all went away. But so did Autonomy as a consumer search player, which speaks volumes that the pattern matching it hawked wasn’t that useful for consumer search. Instead, it was Google that appeared from nowhere pushing link analysis as the more useful technology.

Autonomy, of course, thrived in the enterprise space (where links are less important, and textual analysis more, since you don’t have people purposely trying to alter search rankings). Autonomy still seems to be keeping the IDOL technology for its core business, enterprise search. The real big question is what happens with the Virage video search technology?

Video Search Challenge Isn’t Speech Recognition, It’s Content Owner Management from me in February covers more about how Autonomy purchased Virage, which seems core to its enterprise video search offerings. Blinkx doesn’t seem to be getting Virage or the technology there.

That leaves me wondering if after spinning off Blinkx, could Autonomy down the line decide to go back into the consumer video search space with the Virage technology it retains?

Reed Elsevier is one of the largest, most influential publishers in the world, a powerhouse in the science and medical, legal, education and business markets. But the company also has a major online presence, with more than 1,000 web sites in just its business group.

Last year, Reed Business decided to leverage this significant online presence by creating Zibb, a business-to-business search engine that allows users to search across all of Reed Business’ web sites. But Zibb goes beyond that—Reed also indexes billions of other business-related URLs from all over the web, including competitor web sites.

The result is a very powerful B2B search engine that often returns much better results for business-related queries than mainstream search engines.

Reed Business also decided to integrate this technology into the site search tools for its web sites, which range from the Asia Food Journal to entertainment bellwether Variety. Using search technology from FAST, Zibb on Demand allows each of these sites to offer their own content and the best of the web that’s related to the site’s specialty.

To oversee these efforts, Reed Business hired Stephen Baker to be CEO of Reed Business Search. Stephen is a long-time industry veteran, having put in stints with FAST during its AlltheWeb days, on to Overture and Yahoo, and then back to FAST as general manager of enterprise search. So he’s the ideal person to drive a search group that covers a vast vertical slice of the web, as well as content from a huge global enterprise.

I caught up with Stephen this week to find out what’s new with Zibb and talk about the opportunities he sees going forward in B2B search.

Q: The companies that measure traffic say that Google continues to take market share. Why decide now to start a specialized B2B search service? And why the name Zibb?

Why Zibb? It’s the name of one of Reed Business’ leading publications in the Netherlands. We were looking for a name that is memorable and avoids language-boundaries, which Zibb does. Also, brand names that begin with “z” are more memorable and it has the “BB” in it which could lead to some fun branding and logo design.

While Google is taking market share, we believe that there is a segment of the B2B market that is not 100% satisfied with the results they get from a “general” SE. Specifically, Outsell reports that approximately 1/3 of B2B professionals do not find the information they are looking for when using Google. Our research confirms this.

Furthermore, when you look at the amount of time that professionals spend searching, gathering, and analyzing information, you see that a lot of time is wasted “searching” for information. Between 2001 and 2005, the number of hours spent on information-related tasks increased from 8/week to 13/week and the percentage of time spent “searching” for information increases from 45% to 55%. All of these factors combined indicate that a) search is very important in the daily life of a B2B professional and b) they are not always finding what they are looking for at Google. We feel that this need can be best addressed through a B2B vertical search service.

Q: Do you think all this wasted time and effort is a reflection on the skills of searchers?

Yes and No. Certainly, based on our research, utilization of advanced search features on the major engines is low so when searchers are looking for B2B content, they are often confronted with noise in the results that could be eliminated, to a degree, through advanced search functions. However, this behavior points to the underlying expectation that advanced search features shouldn’t be required to find relevant results. So, with Zibb (and other vertical SE’s), we are basically trying to give the searcher what they want: the “right” answer within a narrower context.

Q: Do you plan to add advanced search or query refinement tools to maybe help people zero-in even more on their target?

Since we’re really focused on improving search productivity and it is our top priority is to provide relevant results with the search and browse paradigm that characterizes Zibb, currently. However, we have a tremendous amount of enriched data describing the content that we aggregate that naturally lends itself to advanced features for the power searcher. So “yes”, over time, we will bring some of that meta data to the forefront of the user experience.

Q: You’re using FAST technology to crawl the web, but also massaging the data you get. You’re crawling 4-5 billion documents every couple of months -What kinds of things are you doing to winnow out non B2B content from search results? And if you’ve got one big index, how and why do search results differ on Variety vs. a site like Flight International?

We are using a series of heuristics to identify relevant B2B content. These range from editorial processes – we have some of the best subject matter experts in house and they continually help us find sites and content that are relevant to their sector – to offline analysis of link structures and anchor texts to identify relevant sites during index build. Of course, we are using an adaptive framework to train the crawler to get smatter with each crawl iteration.

Once we have discarded the non-B2B content, we categorize the content into one of 26 industry taxonomies and describe the content as being a news article, supplier listing, etc. The API’s for the Reed Business Search platform allow for individual websites to specify the query recall set at the industry, sub-industry, or content type level. Hence, Variety can consume only media-related search results while Flight International consumes aerospace results.

Q: Why is Reed Business including content from competitors in Zibb search results?

The goal of Zibb is to provide a search utility for the B2B market. In order to truly provide a valuable service, we believe that Zibb must be a) contextually relevant to the B2B sector and b) comprehensive across all relevant content sources.

Q: I would think some of your competitors wouldn’t want to be part of this—do you have any opting out, or putting other requirements on you to allow you to use their content?

Yes, we occasionally get calls from competitors asking why we are crawling their site so regularly and thoroughly. Some have even blocked our crawlers, however, once they learn that Zibb is focused on becoming a B2B search destination focused on all relevant content, not just Reed Business content, and that our ranking algorithm is unbiased towards Reed Business content, they are usually happy to participate.

Q: How about some stats? How many companies indexed? How many topics, geographic entities and so on?

Currently, we track 17K public companies serving B2B markets, and about 23% of those have strong coverage in our indices. “Coverage,” for our purposes, is a measure of web presence of a particular entity.

Down deep in the tail, we track 1.8M companies also serving B2B professionals, though often in a local or brick and mortar fashion and we’ve found that their online coverage is much weaker with only about 3% passing our “Strong coverage” threshold. Bear in mind that we still track and recognize all of these companies against our full index, we just recognize that many are not well represented online.

Our product categories have 46% strong coverage for the 181K we track, and our topics have 70% strong coverage for the over 2K that we have publicly released to date.

What we’ll see happening over time is that we’ll grow our universe of public and private companies tracked and simultaneously we’ll continue to seek additional web sources of information to strengthen the coverage. This same strategy will be methodically applied to product categories and topics. With our human and machine power we feel that we have a sustainable, scalable, ever-improving approach that will expand even faster as we build within the B2B community who in turn will start helping to shape our growth.

Q: Do B2B searchers differ in how they search from people doing more general searches on Google & Yahoo?

We’re finding that search behavior is relatively consistent. People have the same information needs in B2B as the do in B2C: Read the news and get updates on products, services and companies; perform research functions; look for places to buy things. Of course, search behavior within each of these activities varies (for example, people prefer to browse for news “tell me what’s new” and employ geographic/parametric search for vendors “I’m looking for a company that makes ball bearings in NJ.”) But, in terms of the number of terms per query or the specificity that is employed when looking for eg, product information, there isn’t too much variation.

Q: I would think B2B searchers might tend to have more long tail queries—more specific in expressing what they’re looking for. Have you seen that?

Yes, this is absolutely the case. By nature, B2B trade media is really about having deep subject matter expertise. This is reflected in our content and the needs of our search customers. A portion of the queries that we see are for specific products, parts, and company names which tend to be long-tail queries.

Q: How does Zibb differ from Zibb On Demand?

The Reed Business Search team has built two products. The first is a branded B2B search destination, called Zibb. It is located at www.zibb.com and its Dutch counterpart is located at www.zibbsearch.nl. Zibb on Demand is the hosted vertical search service that we provide to our sites, partners, and affiliates. Zibb on Demand has all of the same features as Zibb and allows the customer to specify the industry and type of content they want included in the results. You can see a couple of examples: Variety.com uses Zibb on Demand to power search across the site, media news search, and media web search. Expert Business Source does the same for small business information.

Q: Customized verticals have been gaining a lot of popularity recently—Google custom search, Eurekster’s Swikis, Rollyo and others. Any plans to create APIs and let users create their own customized B2B vertical search using the Zibb index?

Absolutely. Right now we are focused on building a search platform that meets the immediate needs of our brands, businesses, and partners, however, personalization and custom verticals are features that the Reed Business Search platform support and their deployment will be prioritized as the business needs dictate.

Q: What opportunities do you offer for advertisers who want to reach B2B searchers?

Right now we are using Google AdSense for Search. Later this year, we will offer additional advertising products.

Q: Any tips on optimizing content for sites that want to rank well in Zibb?

We’re looking for unique, targeted, well described content for the B2B sector. You can see the vertical industries that we are targeting on the Zibb.com homepage. If you are writing content targeted to the B2B audience, keep the industry sector in mind and choose relevant phrases. We’re also very good about looking for new content via site maps, RSS, and other natural web protocols. Other than that, if you’re site is well optimized for Google, Yahoo!, MSN, etc., then you’ll do fine in Zibb. And, of course, we have editors on the back-end making sure that our spam detection and categorization algorithms are performing well.

Q: What kind of traffic are you getting? What kind of growth? What growth are you projecting/expecting over the next 1-3-5 years?

So far, both growth of Zibb.com and adoption of Zibb on Demand have been strong. Traffic is growing in excess of 50% month-over-month and Zibb on Demand is currently live on about 30 sites with many, many more scheduled to deploy this year. Looking out beyond 2007 is anyone’s guess. We certainly see a market opportunity for providing a unique B2B search experience and hope that we have a meaningful position in the market beyond 2007.

Thanks, Stephen.

The Q&A Land column appears Wednesdays at Search Engine Land.

News.com reports that Google is expected to release a new upgraded version of their Google Enterprise Search Appliances today. The new Google Mini will “enable companies to perform searches on intranets as well as internal business applications.” The search boxes are expected to be available for purchase as soon as they are announced. I will update this post when I hear about the product launch.

Update: Official press release just released. Updates to the Google Mini include:

(1) Secure Search (2) Google OneBox for Enterprise (3) Site Search Improvements

More details at mini.google.com.

A Google press release from yesterday announced that Google has added a few features to the Google Search Appliance. The new features clearly labeled include:

• Clustering of the results
• Source Biasing Options – define which sources are more authoritative than others
• Add languages such as French, Italian, German and Spanish and Portuguese for query stemming and synonyms dictionaries
• Auto-generate a sitemap for Google.com’s crawler
• Added security for NTLM version 2.0 authentication/authorization
• Open Source Connectors for Microsoft SharePoint

More details here.