When Google launched the Official Google Blog back in 2004, it started as sort of a ho-hum event. There wasn’t anything particularly gripping, and some wondered if the company should be more edgy with its posts. Since then, Google has launched more than 70 additional official blogs over the years. Some have gotten edgy; a few even have comments, but most important, they’ve turned into an essential communications vehicle for the company.

A few weeks ago, I sat down with Karen Wickre, who aside from her formal title at Google of senior manager, global communications & public affairs, is who I’d best describe as "mother of the blogs." Karen is ultimately responsible for how they all work, and she shared some insight on how Google makes use of blogging. Will the big three of Google — Larry, Sergey and Eric — ever blog? Will comments come to the main Google blog? Will Google’s blogging replace press releases? Read on….

Danny: How would you characterize your role in relation to the blogs?

Karen: I shepherd them. I call myself the managing editor, and I’m the gatekeeper for the new blogs. I’ve been with Google a long time and have worked in publishing a long time, so I can’t help but pay attention to the words — to the ways in which Google is communicating with the world.

Danny: I’ve tended to think of you as mother of the main blog and the various blogs in general. Fair to say?

Karen: Mother of the blog is as fair as anything else. [Former PR staffer] Nate Tyler and I pitched the idea of a blog to our team and got agreement to turn it on in May 2004. This was after we’d announced the IPO, but before we went public.

[NOTE: Posts to try out the blog went up in April 2004 but were not made public until after the first official post in May. The very first post seeking job recruits has since been deleted].

Initially the agreement was one post per month. There was no opposition to the idea, but our lawyers literally combed over every word, since we were then in our quiet period. I would say do not launch a blog during a quiet period — or if you do, expect to work with a lot of lawyers on every word!

I love editing and making things read better. That’s what an editor does. A blog is a publication, after all. The idea that I could guide product managers and other Googlers to get new topical blogs off the ground, for example, is very appealing.

Danny: Google has so many blogs now. Do you have to read through each post before they go live, or do some of the blogs have more freedom than others?

Karen: While it’s important to have a review, I never want to overwrite what a Googler is saying about their topic or product. All posts are reviewed by a few relevant people on the immediate team, plus a PR person for approval. As a rule, this isn’t labor-intensive or overbearing. We try to encourage original perspectives and stories insofar as company blogs can feature those. We share drafts in Google Docs and do edits there. Again, I try hard not to overwrite or have the team wordsmith to death. That’s not going to get us interesting reads.

We have different categories of blogs, and the type influences how reviews work. There are product blogs [like the Google Reader Blog], developer blogs [like the Google AJAX Search Blog], country-specific blogs [like the Google Australia Blog] and some vertical interest topics [like the Google Public Policy Blog]. All posts go through someone for approval.

The product blogs are the biggest group, and we have lots of people interested in posting and writing up stuff. With the developer blogs, there tends to be much less editing. There’s no sense wordsmithing there. Developers are a natural blog-reading constituency. In some cases, posts are so well-written from the start, and have a PR staffer on the blog team, such as the Google Public Policy blog, that there is very easy collaboration.

Danny: It feels like Google is running more official blogs than most other companies. Any idea if this is the case, if Google is somehow unique in the amount of blogging it is doing?

Karen: We do have more than 70 blogs now. I feel like Google is one of the few places doing this differently. There are other companies with many blogs, but those tend to be unofficial blogs written by individual developers, such as those from Sun, IBM and Microsoft.

Our network of corporate blogs are definitely all company-based, not personal communications, but we aim to have lots of Googlers writing and covering lots of topics in interesting ways.

Of course, in addition to our many official blogs, we’ve also got individuals like Matt Cutts [from Google's web search quality team] posting. We’d love more like those.

For us, the blog platform is a fast way to publish news and notes about Google and to directly reach millions of people. It’s so fast and easy compared to newsletters or, God-forbid, press releases. It is a PR platform, but we try hard to make it not traditional PR-like. That’s why we want our individuals from teams actually "in the trenches" to do the bulk of the writing.

Danny: So are the blogs working well enough that you might give up press releases?

Karen: Press releases are not going away. There are legal requirements and business requirements for press releases, so they serve a function. Down the line, maybe the whole industry will adopt blogs for public communications. This isn’t up to Google to determine, of course. But we’ve never been a company that issues a ton of press releases.

Much more often than a press release or a Google Gram [these are special email alerts sometimes sent to selected reporters], we’ll issue a blog post.

[Note From Danny: Indeed, over the past two years or so, product coverage I've done often revolves around when a blog post will go up announcing it, rather than a press release being issued. Google often will tell me and others, "and the blog post will go out on...."]

Danny: How many readers do the various blogs have?

Karen: The main Google Blog has over 500,000 [this was in mid-August -- now it is nearly 700,000 readers]. After that, the Google China Blog has many readers, as does Google LatLong. Anything to do with geo stuff is always hugely popular!

NOTE: Karen emailed this week that she also tallied up everything at the end of September using a combination of Google Analytics and FeedBurner. Across the blog network, Google had 16.6 million readers in the first 9 months of the year. She also sent this list of the blogs with the most subscribers according to FeedBurner:

1. Google Blog: 724,522 subscribers
    
2. Google Reader Blog: 79,553 subscribers
    
3. Orkut Blog [Portuguese]: 45,956 subscribers [Orkut is huge in Brazil, and this is reflected in the fact that the official Orkut blog at blog.orkut.com is in Portuguese, while English posts about Orkut are relegated to en.blog.orkut.com].
    
4. Google Japan Blog: 19,781 subscribers
    
5. Google Webmaster Central Blog: 19,926 subscribers
    
6. Google Code Blog: 19,257 subscribers
    
7. Inside AdSense: 16,301 subscribers

Danny: Is there any attempt to try and coordinate postings across the blogs, what people are blogging about, or set any schedule or number of posts to happen?

Karen: We do coordinate posts when we have news that touches on different areas. For example, a Google.org initiative that ties in to our corporate environmental projects gets posted on two blogs, but each one is different. We don’t duplicate posts. A post that’s about YouTube policy may run on the YouTube blog and another one on Google blog.

We do encourage a focus on more interesting items and a regular frequency and not just an endless stream of a product updates and how-to info. So we always aim for useful information and a good read. We don’t have a quota system. We do encourage frequency, but writing something just to have a post doesn’t work.

Danny: How do you work with those teams to encourage more quality posting?

Karen: First, a team has to propose the blog and really understand what’s required to maintain it. A PR "designated hitter" has to be involved and work with the product team to get a product blog going. We’re all in agreement by the time it’s live. There have been some country blogs where there have been one or two interested people running it. In a smaller market, that would be OK. In a bigger country, we would need more people to contribute.

Danny: Sometimes a post on one of Google’s sub-blogs also runs on the main blog. Is there competition to be "good enough" for that?

Karen: Definitely. Someone will write me and say, "I don’t know if this is big enough for the Google Blog." I have a wide-ranging readership in mind, which is global for the Google Blog, and so I ask if the post reflects the multiplicity of Google enough. Things that are small or a feature update on some product probably won’t go on Google blog unless we want to elevate its visibility.

Danny: First the Google Librarian Blog, then Google Webmaster Central Blog gained comments earlier this year, something some argue is required for a blog to call itself a blog. How has that been going? Will more blogs gain them?

Karen: If the team is going to be vigilant about watching them, sure. We want a good read and a conversation, which means watching for spam comments and way off-topic stuff. Developers generally know how to handle comments, so those are easy blogs to enable comments for.

Danny: How about comments on the main Google Blog?

Karen: We’ve been talking about it for a long time internally. Some of us would like to, even though there’s plenty of concern about the time required to monitor comments. We aren’t out to censor criticism, mind you, but to keep the reading and the exchanges useful and informative takes constant review and ability to reply quickly.

Lots of people even outside of Google tell me this is just too hard to manage. After all, big sites like The Washington Post have had problems with comment abuse. So we continue to watch, and evaluate, for the Google Blog itself. Some of our more targeted blogs with smaller readership following very defined topics have an easier time monitoring to maintain an interesting exchange.

Danny: Google News & Finance, Google Maps & Google Earth and the Google Public Policy Blog are all new blogs that have gone up in the past few months. What’s next?

Karen: We may have a few more coming from Google Japan [Google Analytics Blog Japan went up since we talked; over in China, one for webmasters has also been launched]. Perhaps one on Picasa, to cover tips on working with photos [Google Photo Blog went up last month].

Danny: I’ve joked in the past that I can never remember the address of googleblog.blogspot.com for the main official blog, much less google-latlong.blogspot.com for the Google Maps & Earth Blog, Google Lat-Long. Will we get a better address?

Karen: We do want to move to blogs.google.com eventually. We’re still working out how to do it with Blogger (Google’s own blog authoring tool, that it uses for its own blogs).

Danny: Are all the blogs on Blogger? Isn’t Google China Blog not using Blogger?

Karen: We use Blogger wherever it is offered. In China, we don’t offer Blogger as a consumer product, so we publish our various Chinese Google blogs within Blogger and use a third party host.

Danny: Has anyone said they want to use a different blogging platform?

Karen: I don’t think that’s ever come up. Googlers with their own individual blogs can use whatever they want [Matt Cutts uses WordPress, for example]. But we want to showcase Blogger as much as we can.

Danny: How about the Google founders Larry Page and Sergey Brin blogging, or Google CEO Eric Schmidt?

Karen: The big three? I don’t know. They’re so busy. We’d certainly accommodate if they wanted to. They do, however, appreciate our blogs. I think they appreciate the "direct to you" approach. I’ve worked with them on a few of the posts that are essentially statements. These are a nice way to state our position so that reporters can work from it directly. It also becomes our standing statement on something. That’s been nice for us in a few instances where we’d never put out a press release.

The Google bombing phenomena is an example of this. I think it was almost two years ago, we put out an explanation of what this thing was. That has had huge legs in the blogosphere. People have pointed to it, linked to it and said "Here’s the definitive statement on this thing."

Danny: What’s are the most popular posts you’ve done?

Karen: Anything to do with Google Earth and Google Book Search has a lot of readers. And our funny bits on the main Google Blog, like the explanation about the Valentine’s Day doodle or 100 pounds of Silly Putty.

NOTE: Karen has since sent me this comprehensive list:

1. Strawberries are red, stems are green, Feb. 14, 2007: 226,131 pageviews [Official Google Blog]
    
2. Pré-visualização da Reestilização: Simplicidade Fiel, Aug. 24, 2007: 117,507 pageviews [Orkut Blog]
    
3. An Update on Google Video Feedback, Aug. 20, 2007: 111,967 pageviews [Official Google Blog]
    
4. Build your own Google homepage, Feb. 13, 2005: 103,701 pageviews [Official Google Blog]
    
5. Doing the Shuffle, June 22, 2007: 80,811 pageviews [Google Reader Blog]
    
6. "Go Go (Reader) Gadget, April 6, 2007: 72,252 pageviews [Google Reader Blog]
    
7. Discover your links, Feb. 5, 2007: 62,985 pageviews [Google Webmaster Central Blog]
    
8. A quick word about Googlebombs, Jan. 25, 2007: 62,640 pageviews [Google Webmaster Central Blog]
    
9. 日本語の Google Earth, Sept. 14, 2006: 61,953 pageviews [Google Japan Blog]
    
10. Breaking up isn’t hard to do, Dec. 18, 2006: 60,229 pageviews [Google Reader Blog]
     
11. Speaking of Summer, Feb. 15, 2007: 35,652 pageviews [Google Code Blog]
     
12. Announcing Tesseract OCR: Aug. 30, 2006: 30,536 pageviews  [Google Code Blog]
     
13. Google releases patches that enhance the manageability and reliability of MySQL, April 23, 2007: 29,339 pageviews  [Google Code Blog]
     
14. Introducing video units, Oct. 8, 2007: 17,197 pageviews [Inside AdSense]
     
15. Get inline, Aug. 21, 2007: 16,272 pageviews [Inside AdSense]

Looking for more about Google and blogging? Last year, Karen did an hour-long podcast on working on the blog you’ll find here. Also see 10 Google Feeds You Should Subscribe To on how to keep up on everything from all Google blogs, if you want.

Seth Godin is widely known as one of the foremost advocates of “permission marketing,” promotional campaigns that don’t rely on interrupting the attention of your customers, but rather engage them and even turn them into enthusiastic advocates that volunteer time and effort to help you promote your products or services.

Godin doesn’t shy away from controversy. He’s tweaked the noses of other well-known pundits, and in recent years has also blasted search marketers, accusing them of a variety of sins. But Godin is no stranger to search: Yoyodyne, an interactive direct marketing company he company he founded and ran was acquired by Yahoo in 1998. One of his current efforts is a human-powered search service (though he wouldn’t call it that): Squidoo.

I asked Godin to elaborate on his issues with search marketing and to talk a bit about his goals and aspirations with Squidoo. Read on for his fascinating views and opinions on what search marketers are currently doing wrong and what they can do to improve their campaigns and create a more effective dialogue with searchers.

Q. A few years ago, you wrote that you thought most search engine optimization wasn’t worth the money. Since then, your stance seems to have softened somewhat, with your acknowledging the work of some pros like Andrew Goodman and Aaron Wall. What’s your position on SEO today?

I think it’s important for everyone to act as if it doesn’t work. When it becomes a crutch, when you use SEO as a replacement for something else, you fail. I do agree that common sense white hat SEO is a smart tool—if you’ve done everything else—and your organization has appropriate scale. Short version: No magic bullets.

Q: What about companies that are operating in ruthlessly competitive areas, like travel, gambling, jewelry, etc? One of Yahoo’s chief spam cops, Tim Mayer, once said “If you’re being entirely organic and going after ‘Viagra,’ it’s like taking a sword to a gunfight. You just aren’t going to rank.” After all, there are black hats out there—shouldn’t you be realistic and compete with the same techniques they’re using?

Sure, travel is a brutally competitive area. And if you treat it like a commodity, you’ll get treated like a commodity in return. My point is that a tough business is not an excuse to cheat or to rely on tactics that don’t last.

If I ran a travel site, I’d engage my best customers to build blogs and Squidoo lenses and to use Digg to point to reviews and insights and things that would make people WANT to seek me out.

Spending money on ads or commodity-focused SEO is the last gasp of someone who is short on innovation, imagination and great stuff!

Q. At SES New York you said that people were attending the conference because “search is broken.” Can you elaborate on that?

Well, if search worked, then you wouldn’t need a strategy! People would find you when you needed to be found, and find someone else the rest of the time. Of course, search is always going to be a bit broken (though it keeps getting better) and the more human person to person recommending that gets included (including squidoo.com), the better it’s going to work.

Q: You describe Squidoo as “1) thousands of people creating a handbuilt catalog of the best stuff online 2) a free and fun way to make your own page and get traffic 3) a place to find what you’re looking for, fast. How does Squidoo differ from other community-built sites, like About.com, Wikipedia or others?

Wikipedia has one voice. Every contributor is anonymous. About.com has a few hundred voices, and only page on a topic. We’re a cross between the two, but we add in 65,000 people building the lenses… so you can find the people who you trust, the judgment that you rely on, and getting something more focused and up to date than about.com.

Q: I’ve found Squidoo to be helpful, but have also searched for some things where nobody has yet built a “lens” (focused topic page) on the subject. That seems to be a problem with all people-driven search projects—they don’t scale. How will you deal with this?

Well, they do scale (Wikipedia is xxxx times as big as the Encyclopedia Britannica) but not as fast as you (or I) might like. The win happens when each lensmaster brings in ten more. Do that three times and you’re 1000 times bigger than you were.

Q: I’ve also been critical of people-driven search because it’s often hard to tell whether someone is trustworthy or credible in their recommendations. How do you address these issues of trust and reliability?

You’re exactly right. You can’t tell. You can’t tell if a Squidoo lens is legit, or if Wikipedia got hacked. But as long as a) it’s better than the alternatives and b) improvable and c) another point of view then I argue that it’s worthwhile.

Blogs aren’t reliable either, but I’m glad they exist!

Q. If someone were just starting out (or starting over) with a web site, what key things would you recommend they spend time doing to attract visitors and engage with customers online?

Far and away job one: Make it easy for your best customers to tell their friends. Make products worth blogging about, sites that host consumer content that they promote, affiliate programs they want to embrace. Build the entire site about permission and not a pretty edifice.

Q: The Wall Street Journal recently ran an article about companies that have sprung up solely to help promote web sites by encouraging people to email articles to friends, prodding them to vote for content on sites like Digg and so on. This is a close cousin of what you’re recommending, but it’s also somewhat artificial. Do you think this is a legitimate approach?

I hate this.

It’s another reminder of the selfishness and lack of passion that marketers often show.

Q. You’ve said that many search advertisers miss the boat by sending searchers to landing pages that try to get them to buy or convert immediately. Instead, savvy advertisers develop an “offer culture” where they experiment with landing pages and create a step-by-step process that encourages participation rather than a single event. While this sounds great, what about searchers who want to make an immediate purchase?

I think what I meant is that many advertisers make one of two common mistakes: a) they get lazy and send everyone to a generic page or worse, to a home page, instead of customizing an offer to match the ad itself, or b) they get impatient and try to turn that first click into a sale right now. While that works sometimes, we all know that real sales take time, and the beauty of permission is that you can educate and follow up later.

Q: Many search marketers are small business people—mom & pops or individuals working the web part-time. That’s what makes search appealing to them—it’s a relatively efficient way to reach many customers without spending a lot of time or money. Permission marketing seems more like a process that takes time and perhaps additional resources to follow up after the first contact. What suggestions can you offer for search marketers who have limited time or resources to adopt this approach?

I think if you’re not planning on being around in three years, you’re right, don’t build an asset that grows over time. Slowing down is sometimes the very best way to speed up.

Q. You wrote of the rumored Yahoo/Microsoft merger, “The best things to ever come out of Yahoo, as far as I’m concerned, have been the work of individuals. Not of some hyperbolic purple and yellow machine, but from people, strong-willed individuals willing to buck the bureaucracy.” Can you give some examples?

Yahoo Mail, Zeitgeist, Yahoo Finance, Flickr… these were all fairly renegade efforts, some internal, some not. Compare them to Yahoo Auctions, which was a strategic, top down, carefully focused effort to beat eBay. (Editor’s note: Yahoo recently announced that it was terminating auctions, working with partner eBay instead.)

It’s not just Yahoo, of course. Most organizations work this way.

Q: Yahoo just changed its mission statement—it’s now “To connect people to their passions, communities, and the world’s knowledge.” Microsoft’s mission is much vaguer: “At Microsoft, we work to help people and businesses throughout the world realize their full potential.” Do you think the two companies would make a good fit?

I don’t think the missions have nearly as much to do with a match as the people. And I worry about the people fit.

Q: Crystal ball time: Describe the world of the web 5 years from now from a marketer’s standpoint. Will people still be using search engines as heavily as now, or will there be new ways we seek out information and products online? Will there be a shift away from traditional types of marketing to new ways of engaging with customers online? And who will the major players be?

I think there is going to be a major shift on a number of fronts.

First, permission marketing is really going to hit home when advertisers realize that the only ads they can deliver are ads that people sought out (at least among the most desirable consumers).

Second, anonymous media is going to take a real hit as we discover that lots of anonymous people posting movies and comments and such is a real pain in the neck.

Third, RSS and similar private channels will become much bigger.

And fourth, flipping the funnel and talking through the converted will elect presidents and sell lots of products.

Thanks, Seth.

The Q&A Land column appears Wednesdays at Search Engine Land.

Search engines are great tools to help you find things you suspect exist on the web, but they leave a lot to be desired when it comes to discovering the unexpected or unknown. Sure, you may find something really cool after doing the result-page/back-button tango for a while, but that’s an inefficient approach.

Enter StumbleUpon, a service that lets you “channel surf” the internet, displaying sites that others have “stumbled upon” and recommended. StumbleUpon is a hybrid: Part directory of human selected sites, part search engine (the millions of pages recommended by “stumblers” have been fully indexed and categorized behind the scenes), part social bookmarking service. It’s all of these things, and yet offers a unique web discovery experience unlike any other I’ve seen.

In his first “Let’s Get Social” column, Forget ABCs – The Social Media Alphabet Is DNRS, Neil Patel introduced StumbleUpon. In today’s Q&A, I asked Garrett Camp, StumbleUpon’s Founder and Chief Architect, to take us behind the scenes of this intriguing discovery engine.

Q: You started StumbleUpon in November 2001 – a pretty dark time when lots of web sites & online businesses were crashing and burning. What inspired you to start at that time? And why did you see a need for this type of service?

We realized that search engines are useful when you know exactly what you are looking for, but for discovering new and personally interesting online media keyword search wasn’t always the best option. We created StumbleUpon so people can “stumble upon” sites that have been submitted and rated by like-minded people, rather than presented with the most popular sites for a given keyword. We designed StumbleUpon to blend collaborative human input with machine learning techniques, so users could discover great content they wouldn’t have thought to search for. We’ve been focused on social media and collaborative discovery long before the phrase “Web 2.0″ became popular in 2004, and now have over 2M members discovering and sharing interesting web content.

You’ve said that “stumbling” is more a process of discovery than search—you’ve even called StumbleUpon a “remote control” for the web. Can you describe how StumbleUpon resembles “channel surfing” and how you customize the experience for each user?

Search is great for textual content or if you know exactly what you want, but if you’re simply looking to browse through relevant and high-quality web content we think the “channel-surfing” metaphor makes more sense. Instead of having to search, scan, click, go back, and repeat, you simply surf directly between relevant content which has been given the “thumbs-up” by friends and other web surfers with similar interests. We have nearly 500 topics that users may choose to indicate preferences, so your recommendations become precisely tailored to your interests. Essentially we combine community reviews and social feedback with clustering and evolutionary methods to create a personalized recommendation engine. This helps people discover interesting websites, photos and videos recommended by their friends and like-minded people.

You’ve got about 9 million sites indexed—how dynamic is that index as new content is added to the web or old content disappears?

We like to think we have the best of the web in our database. Our 2 million registered users stumble around 5 million times a day, so we have a pretty active user base. If they find something new, it’s incredibly easy for them to submit it to us. All they need to do is click the thumbs-up button on the toolbar and it’s submitted to our database. We get over 16,000 new URL submissions a day – all new and unique content endorsed by our members.

I’ve been skeptical of many aspects of social search. StumbleUpon incorporates a number of social search techniques (thumbs up-or-down voting for web pages, tagging, collaborative filtering and so on), and yet also does a lot of algorithmic heavy-lifting in the background. What kind of indexing and classification are you doing in addition to observing user behavior?

We have a classification engine which automatically places content into one of 500 predefined categories based upon on-the-page factors. This means most content submitted can be distributed to interested members even before tags have been applied. We also have the entire database indexed so you can stumble by keyword if you want to get fairly specific… for example stumbling through just pictures of “sunsets” or discovering new “fonts”. We use classification and indexing in combination with social networks and user ratings to improve the quality of recommendations and coordinate the distribution of relevant content.

StumbleUpon offers a lot of community features. You can browse the StumbleUpon community to find users with similar interests and even become “friends” with them. You also encourage people to share a lot of information about themselves. Does the user experience change depending upon how much self-info you’ve provided, or the network of connections you’ve made? How?

Your stumbling experience improves as you get more involved in the community. Once you add a picture and start reviewing pages, you’ll start meeting other community members who like similar types of sites. We’ve found that most active members will upload a picture or fill out their favorite music, movies etc… so they can start meeting like-minded people. Over time the social aspect of StumbleUpon makes it more compelling, since you’ll discover pages your friends have recommended and learn about their tastes as you stumble.

What about people who aren’t comfortable sharing personal information? What kind of privacy measures do you have in place?

We have a few privacy features, and how much information you share is completely up to you. If you want to remain an anonymous member without a public presence simply don’t upload a photo or fill in your profile. You can also choose to not share your ratings if you like, but we’ve found the vast majority of people don’t mind other people visiting their profile or reading the reviews they write.

One of my favorite features of StumbleUpon is the automatic addition of StumbleUpon ratings and a link to reviews in search results from Google, Yahoo, Ask, AOL and Microsoft. This adds a “second layer” of relevance to the results, showing you at a glance which pages have been given a thumbs-up or good reviews by stumblers. The StumbleUpon toolbar also lets you search by keyword, but sends you directly to a web page rather than showing search results. Why take that approach rather than showing a list of the top pages StumbleUpon has indexed for a particular keyword?

When developing the Search Reviews feature, we figured that most people didn’t want to switch to a different search engine, and would rather just have social aspects added to their current search engine of choice. So by adding links to Google, Yahoo et al, you can still query a multi-billion-page index, and get additional social metadata when simple queries are submitted. This means you don’t need to change your search behavior at all to enjoy the benefits of community endorsements when they exist. As for Stumbling by keyword, it’s really to just keep the navigation experience consistent, and maintain the sense of serendipity.

We’ve seen small but growing traffic at Search Engine Land from StumbleUpon users. Any recommendations for search marketers wanting to tap into the StumbleUpon community?

StumbleUpon has a unique business model that works well for marketers where we can deliver traffic directly to your site. You can target by category, age, gender and location. So for product launches, distributing audio/visual content or just getting feedback on your blog, StumbleUpon often works better than PPC approaches since targeting is precise and no click through is required. More information on opportunities is available on our advertisers page.

Talk about your future plans.

We are always looking to improve the user experience and the technology behind StumbleUpon. We are currently focused broadly on both social search and video discovery, as well as algorithmic improvements to our recommendation engine. We are also re-designing our entire site and investigating bringing StumbleUpon to other platforms beyond the browser. Stay tuned…

Thanks much, Garrett.

The Q&A Land column appears Wednesdays at Search Engine Land.

Exalead is one of those great search engines that most people haven’t heard of. And that’s a shame, because it offers some really powerful and unique features that most information professionals love.

Exalead is also the name of an enterprise search company, co-founded by today’s Q&A participant, François Bourdoncle. Working on web technologies from the very early days, he co-founded Exalead in 2000 with the goal of revolutionizing the search engine software market by providing users with a unified technology platform to access information in the enterprise. Exalead.com, the web search engine, showcases the company’s capabilities in search.

Francois and Exalead also play a key role in the development of the government-funded pan-European search engine Quaero, which has had its share of controversy. Read on for our Q&A…

You have a long history in web search, having worked on the LiveTopics search refinement tool for AltaVista back in 1996-97, among other things. What are the most notable changes you’ve observed in the web search world over the past decade?

I am intrigued by the sheer number of searches conducted daily. The volume is a reflection that search has moved to the center of all we do online and, for that matter, offline. Using a PC has changed from simply creating a document or some other file form to a process that revolves around search. We’ve seen the development of search-centric computing and search-based online activities.

Further, we’re seeing search evolve from a consumable process, that is entering a keyword or two, running the query, finding it and moving on from there. Search activities are increasingly involving the creation of a temporary – or long-term – storage or archiving of the search input and findings.

Lastly, there is a negative development we’re seeing, one that everyone is aware of, if not experiencing: frustration. Finding information continues to be harder than it should be. People quickly lose patience if they can’t find what they’re searching for right away, yet most search engines essentially ask people to scroll through lists and lists on pages and pages of results, without any automated organizational assistance.

Exalead offers a lot more information on search result pages than other search engines, including thumbnail images, and an option to display other content from a site including directory listings, audio, video or RSS feeds. Are your users interacting with these additional details?

The short answer is: without a doubt, yes. These features are what set Exalead apart from other Web search engines, and help to increase our customer loyalty and usage. Over the past 18 months, most recently in New York, we’ve been conducting intensive research studies to gain insight into users’ search habits and preferences.

What we’ve found is that the thumbnail images Exalead offers for all search results is the most popular feature. We’ve designed the user interface so that users can turn this feature on or off, but the feedback we’ve received is that the ability to search visually is a huge advantage. The Internet is second-nature for most people today, especially the younger generations, as young as high school students. They don’t like to be slowed down by having to click through to a new browser window to see results.

The way we also present “in-context” audio, video, and RSS content makes it very easy for novice users to find multimedia content on the Web.

Exalead uses Open Directory Project data for directory listings, and it also looks like you give a relevance boost to sites included in the ODP. A lot of people have said that the ODP is walking dead now, with little if any development or maintenance of the directory. Have you found that to be the case?

We use ODP as a way to categorize search results in a richer way that what our automatic classification technology (Related Terms) provides. This has proven very useful for our “Search By Serendipity”™ search experience. It is true that ODP has been less frequently updated than it used to be, and that social tagging sites are now being used as a way to organize the Web. We are working on this new trend actively and will introduce new products later this year.

I’ve been critical of tagging, because most people apply tags carelessly. Tags on the web lack a controlled vocabulary and aren’t based on disciplined ontologies, so they can be ambiguous or even misleading. How do you remove the “noise” from tags to make them useful?

I am not sure that tags and disciplined ontologies serve the same purpose. Disciplined ontologies are best used for organizing “long-lived information,” whereas tags are best used for quick & efficient organization of “short-lived information,” such as project-related documents, bookmarks, etc. Therefore, tags offer an interesting trade-off between ease-of-use vs. efficiency for finding relevant information. An interesting example of a mixed approach is Wikipedia, which uses semi-controlled tags that are very similar to ontologies, with the exception that they are not hierarchical.

Your advanced search offers some great features no other web search engine offers – things like phonetic search, approximate spelling, proximity and so on. Other search engines say they don’t offer these because they are costly and searchers don’t use them. Why does Exalead have them?

Our raison d’être is to make search easier, less frustrating and more efficient for everyone. Our advanced features help make searching on exalead.com a more satisfying experience. Both novice and so-called power-searchers use these features to serve their own purposes, to their own advantages. The notion that those features and functions drive up the cost is bogus; expense management occurs at the core infrastructure level, not at the functional level, which would include advance search features. Also, using an enterprise search technology to power a Web search engine is the easiest way to provide these features efficiently.

Your “about” page states: “It also adapts to user habits for a uniquely satisfying and efficient search experience.” Does Exalead do personalization or tailor search results for individual users? If so, how does this work?

Yes and no. Right now users can personalize their home page by creating shortcuts, selecting a language preference or by deciding how they want to view results. In the near future, Exalead will be unveiling new search 2.0 features that take personalization and search organization to an entirely new level. We’ll have more to say on that for another Q&A in the not-too-distant future.

There are two ends to the personalization spectrum: Google’s approach, which essentially observes your behavior and adjusts accordingly, without much other input from you, and Yahoo’s approach, which requires you to explicitly shape your personal web. I like the automated approach but I find it frustrating that Google won’t let me “override” results I personally consider poor. What is Exalead’s approach? And how do you balance personalization with privacy concerns?

Personalization is indeed a very important aspect of search, and one that Exalead is heavily investing in. As usual, we are trying to find the right trade-off between automation and flexibility, and will come up with new ideas for having the “best of both worlds” in our forthcoming products.

What is your or Exalead’s involvement with Quaero, the European search engine funded by governments?

I must note Quaero is one of the most talked-about, least-understood research projects in recent memory. Quaero is a research initiative funded, in part by the French government and, until recently, the German government. Quaero, which in Latin means “to search,” is focused on multimedia search (audio, video, music, images, etc.).

Exalead is one of the four lead companies in the project. These leaders define what the R&D objectives in the project ought to be. Over time, the four leaders will integrate some of Quaero’s technologies into their products, when these technologies become mature enough. Our search technology is also used by other project members.

Despite what some politicians have said, Quaero is not a collaborative effort aimed at creating a “Google killer.” It is a research project on multimedia search technologies, period. In particular, image tagging, speech-to-text and machine translations, etc. It may very well produce technology or technology advancements that improve search, but Exalead is pursuing its own course of product development and go-to-market activity. What comes of Quaero will not determine what Exalead becomes or offers.

Thanks, Francois.

Reed Elsevier is one of the largest, most influential publishers in the world, a powerhouse in the science and medical, legal, education and business markets. But the company also has a major online presence, with more than 1,000 web sites in just its business group.

Last year, Reed Business decided to leverage this significant online presence by creating Zibb, a business-to-business search engine that allows users to search across all of Reed Business’ web sites. But Zibb goes beyond that—Reed also indexes billions of other business-related URLs from all over the web, including competitor web sites.

The result is a very powerful B2B search engine that often returns much better results for business-related queries than mainstream search engines.

Reed Business also decided to integrate this technology into the site search tools for its web sites, which range from the Asia Food Journal to entertainment bellwether Variety. Using search technology from FAST, Zibb on Demand allows each of these sites to offer their own content and the best of the web that’s related to the site’s specialty.

To oversee these efforts, Reed Business hired Stephen Baker to be CEO of Reed Business Search. Stephen is a long-time industry veteran, having put in stints with FAST during its AlltheWeb days, on to Overture and Yahoo, and then back to FAST as general manager of enterprise search. So he’s the ideal person to drive a search group that covers a vast vertical slice of the web, as well as content from a huge global enterprise.

I caught up with Stephen this week to find out what’s new with Zibb and talk about the opportunities he sees going forward in B2B search.

Q: The companies that measure traffic say that Google continues to take market share. Why decide now to start a specialized B2B search service? And why the name Zibb?

Why Zibb? It’s the name of one of Reed Business’ leading publications in the Netherlands. We were looking for a name that is memorable and avoids language-boundaries, which Zibb does. Also, brand names that begin with “z” are more memorable and it has the “BB” in it which could lead to some fun branding and logo design.

While Google is taking market share, we believe that there is a segment of the B2B market that is not 100% satisfied with the results they get from a “general” SE. Specifically, Outsell reports that approximately 1/3 of B2B professionals do not find the information they are looking for when using Google. Our research confirms this.

Furthermore, when you look at the amount of time that professionals spend searching, gathering, and analyzing information, you see that a lot of time is wasted “searching” for information. Between 2001 and 2005, the number of hours spent on information-related tasks increased from 8/week to 13/week and the percentage of time spent “searching” for information increases from 45% to 55%. All of these factors combined indicate that a) search is very important in the daily life of a B2B professional and b) they are not always finding what they are looking for at Google. We feel that this need can be best addressed through a B2B vertical search service.

Q: Do you think all this wasted time and effort is a reflection on the skills of searchers?

Yes and No. Certainly, based on our research, utilization of advanced search features on the major engines is low so when searchers are looking for B2B content, they are often confronted with noise in the results that could be eliminated, to a degree, through advanced search functions. However, this behavior points to the underlying expectation that advanced search features shouldn’t be required to find relevant results. So, with Zibb (and other vertical SE’s), we are basically trying to give the searcher what they want: the “right” answer within a narrower context.

Q: Do you plan to add advanced search or query refinement tools to maybe help people zero-in even more on their target?

Since we’re really focused on improving search productivity and it is our top priority is to provide relevant results with the search and browse paradigm that characterizes Zibb, currently. However, we have a tremendous amount of enriched data describing the content that we aggregate that naturally lends itself to advanced features for the power searcher. So “yes”, over time, we will bring some of that meta data to the forefront of the user experience.

Q: You’re using FAST technology to crawl the web, but also massaging the data you get. You’re crawling 4-5 billion documents every couple of months -What kinds of things are you doing to winnow out non B2B content from search results? And if you’ve got one big index, how and why do search results differ on Variety vs. a site like Flight International?

We are using a series of heuristics to identify relevant B2B content. These range from editorial processes – we have some of the best subject matter experts in house and they continually help us find sites and content that are relevant to their sector – to offline analysis of link structures and anchor texts to identify relevant sites during index build. Of course, we are using an adaptive framework to train the crawler to get smatter with each crawl iteration.

Once we have discarded the non-B2B content, we categorize the content into one of 26 industry taxonomies and describe the content as being a news article, supplier listing, etc. The API’s for the Reed Business Search platform allow for individual websites to specify the query recall set at the industry, sub-industry, or content type level. Hence, Variety can consume only media-related search results while Flight International consumes aerospace results.

Q: Why is Reed Business including content from competitors in Zibb search results?

The goal of Zibb is to provide a search utility for the B2B market. In order to truly provide a valuable service, we believe that Zibb must be a) contextually relevant to the B2B sector and b) comprehensive across all relevant content sources.

Q: I would think some of your competitors wouldn’t want to be part of this—do you have any opting out, or putting other requirements on you to allow you to use their content?

Yes, we occasionally get calls from competitors asking why we are crawling their site so regularly and thoroughly. Some have even blocked our crawlers, however, once they learn that Zibb is focused on becoming a B2B search destination focused on all relevant content, not just Reed Business content, and that our ranking algorithm is unbiased towards Reed Business content, they are usually happy to participate.

Q: How about some stats? How many companies indexed? How many topics, geographic entities and so on?

Currently, we track 17K public companies serving B2B markets, and about 23% of those have strong coverage in our indices. “Coverage,” for our purposes, is a measure of web presence of a particular entity.

Down deep in the tail, we track 1.8M companies also serving B2B professionals, though often in a local or brick and mortar fashion and we’ve found that their online coverage is much weaker with only about 3% passing our “Strong coverage” threshold. Bear in mind that we still track and recognize all of these companies against our full index, we just recognize that many are not well represented online.

Our product categories have 46% strong coverage for the 181K we track, and our topics have 70% strong coverage for the over 2K that we have publicly released to date.

What we’ll see happening over time is that we’ll grow our universe of public and private companies tracked and simultaneously we’ll continue to seek additional web sources of information to strengthen the coverage. This same strategy will be methodically applied to product categories and topics. With our human and machine power we feel that we have a sustainable, scalable, ever-improving approach that will expand even faster as we build within the B2B community who in turn will start helping to shape our growth.

Q: Do B2B searchers differ in how they search from people doing more general searches on Google & Yahoo?

We’re finding that search behavior is relatively consistent. People have the same information needs in B2B as the do in B2C: Read the news and get updates on products, services and companies; perform research functions; look for places to buy things. Of course, search behavior within each of these activities varies (for example, people prefer to browse for news “tell me what’s new” and employ geographic/parametric search for vendors “I’m looking for a company that makes ball bearings in NJ.”) But, in terms of the number of terms per query or the specificity that is employed when looking for eg, product information, there isn’t too much variation.

Q: I would think B2B searchers might tend to have more long tail queries—more specific in expressing what they’re looking for. Have you seen that?

Yes, this is absolutely the case. By nature, B2B trade media is really about having deep subject matter expertise. This is reflected in our content and the needs of our search customers. A portion of the queries that we see are for specific products, parts, and company names which tend to be long-tail queries.

Q: How does Zibb differ from Zibb On Demand?

The Reed Business Search team has built two products. The first is a branded B2B search destination, called Zibb. It is located at www.zibb.com and its Dutch counterpart is located at www.zibbsearch.nl. Zibb on Demand is the hosted vertical search service that we provide to our sites, partners, and affiliates. Zibb on Demand has all of the same features as Zibb and allows the customer to specify the industry and type of content they want included in the results. You can see a couple of examples: Variety.com uses Zibb on Demand to power search across the site, media news search, and media web search. Expert Business Source does the same for small business information.

Q: Customized verticals have been gaining a lot of popularity recently—Google custom search, Eurekster’s Swikis, Rollyo and others. Any plans to create APIs and let users create their own customized B2B vertical search using the Zibb index?

Absolutely. Right now we are focused on building a search platform that meets the immediate needs of our brands, businesses, and partners, however, personalization and custom verticals are features that the Reed Business Search platform support and their deployment will be prioritized as the business needs dictate.

Q: What opportunities do you offer for advertisers who want to reach B2B searchers?

Right now we are using Google AdSense for Search. Later this year, we will offer additional advertising products.

Q: Any tips on optimizing content for sites that want to rank well in Zibb?

We’re looking for unique, targeted, well described content for the B2B sector. You can see the vertical industries that we are targeting on the Zibb.com homepage. If you are writing content targeted to the B2B audience, keep the industry sector in mind and choose relevant phrases. We’re also very good about looking for new content via site maps, RSS, and other natural web protocols. Other than that, if you’re site is well optimized for Google, Yahoo!, MSN, etc., then you’ll do fine in Zibb. And, of course, we have editors on the back-end making sure that our spam detection and categorization algorithms are performing well.

Q: What kind of traffic are you getting? What kind of growth? What growth are you projecting/expecting over the next 1-3-5 years?

So far, both growth of Zibb.com and adoption of Zibb on Demand have been strong. Traffic is growing in excess of 50% month-over-month and Zibb on Demand is currently live on about 30 sites with many, many more scheduled to deploy this year. Looking out beyond 2007 is anyone’s guess. We certainly see a market opportunity for providing a unique B2B search experience and hope that we have a meaningful position in the market beyond 2007.

Thanks, Stephen.

The Q&A Land column appears Wednesdays at Search Engine Land.

Over the past decade, I’ve seen a lot of search tools that were supposed to transform my life. Few of them have. But Techmeme was one of those. When it kicked off back in September 2005, I wrote a review, gave it a preliminary thumbs-up and soon found myself addicted. It has become my newspaper, my front page guide to what’s going in the blogosphere relating to tech.

I met Techmeme creator Gabe Rivera in person for the first time last August, during the Search Engine Strategies conference in San Jose. I roped him into being on the Meet The Blog & Feed Search Engines panel at the last minute, where he joined Technorati, Topix, Bloglines and Sphere. Gabe asked the audience if anyone had heard of Techmeme. Apparently, no hands went up — at least in the audience. All the panelists certainly had.

I love that anecdote, because I feel it largely reflects how Techmeme operates. The masses might have Digg, but perhaps the influencers have Techmeme. Certainly plenty of large, influential bloggers I know keep an eye on what it is covering. But I recommend it for anyone, not just influencers, for the easy way it organizes what’s happening with technology stories.

I caught up with Gabe last week, to talk about the service, how it has grown, operates and future plans.

Q. Do you feel Techmeme has a more early adopter or influencer audience?

Yes, and in particular a lot of influencers in the tech media are Techmeme readers.  A good number of high profile bloggers and reporters reload Techmeme all day long in fact.  One sign of which is citations of course, for example the “via” link included the other day on Engadget [link here] and so many other sites. I’ve also run into a number of VCs and executives who use it too, so I’m pleased with that aspect of the demographics.

The traffic is not huge. I still get fewer than 30,000 daily uniques [unique visitors per day]. But then these are actual loyal users, since I see very little search traffic, perhaps due to poor SEO as my permalinks don’t lead to individual articles.

[Note: Techmeme permalinks are little paper icons that show next to stories, marked as "Click This" in the illustration below:

To understand why the SEO is "poor," compare how Techmeme lists the TechCrunch story about the Wikiseek launch to how Digg does it. Techmeme jumps you to where the story is mixed with others on different topics, diluting the overall content or theme of the page. Digg give the story its own dedicated page, which often fills up with user generated content from comments].

But I doubt even good SEO will help a lot. People who arrive from Google tend to be looking for primary documents, not an aggregated page that quotes primary documents, and therefore rarely return anyway.

Q. Have you considered the idea of creating a Techmeme widget or button of some type, similar to those offered by Del.icio.us or Digg, so people could easily point to how they are listed at Techmeme or tell people to see discussions on a topic, the way Technorati or Sphere do?

Given the number of sites I monitor, that would probably only help a small number of bloggers a small portion of the time. So it’s probably not the highest priority feature for me to work on.

Q. How about a way to enter a URL and see if it’s been on Techmeme, to see the related posts that are clustered with a main item? Or a way to keyword search for articles or subjects.

That would certainly be a welcome feature, useful for some.  But I’m really inclined to offer something more powerful if I do add search. In the meantime Techmeme River [launched in December, more background here] actually enables search, if what you’re searching for appeared in the last 5 days.

Q. Personally, I’ve also wanted that ability to go back in time easily and see the “front pages.” I know how to do it, as I’ve written about, but I want it even easier.

I suppose you mean a simple, clickable interface in additional to the “Page version” text input I offer already. I hear a lot of requests for obviously useful features like that. But considering the cost/benefit, I believe there are better things for me to tackle for now. I’ll be sure to write that one down though!

Q. Is Techmeme an echo chamber, just showing blogs commenting about blogs commenting about blogs? Does Techmeme feed into that echo chamber? Or how do you break apart the conversations on a particular topic into sub-conversations or topics?

Clearly Techmeme creates superficial incentives for “echo chamber” participation, yet I don’t see clear evidence that this makes things noticeably worse.  I still like to trot out the example of the day my site launched. eBay’s acquisition of Skype became one of those huge story clusters, and this was hours before Techmeme [then tech.memeorandum] was publicly launched, i.e. before anyone believed they could get on the site by linking to stories.

I’d also point out the idea of many headlines on a single major story is not a problem in and of itself. Consider that the iPhone unveiling will probably be one of the major stories for all of 2007. So on one day for it to account for 40 percent of the headlines on Techmeme is not all that out of whack.

A more serious problem is when multiple stories essentially say the same thing. Ideally no two posts should just be rewrites of the same facts.  Unfortunately it’s hard to accomplish that with software. Now I do expect to introduce something that tends to reduce those story clusters in a smarter way, but it will take some time to do that right.

Q. This page shows how the Yahoo purchase of MyBlogLog broke the story up into various clusters. You’ve got news of the sale, but also a report on the sale price, Yahoo’s Jeremy Zawodny doing a personal welcome. How do you determine which of the many stories to break out into these sub-clusters?

It’s driven by what my software things is the most important story inside of a cluster, then ranked by what it believes are the most important stories that discuss that. On that page, it just thinks that Om Malik’s post was the second most important [after the headline story from the CESblog].

Q. Over time, those clusters change. Some stories rise; all of them eventually fall. How does that work?

All of this derives from this “importance” measure I just alluded to. Importance is determined by a number of factors. Citations can increase importance, so a post that accumulates inbound links can rise. Time is a factor as well. A headline that’s appeared on the page for most of the day loses importance. Headlines usually fall off the page when the time component swamps all other factors. That’s how old news gives way to newer news.

Q. Old meaning how long from when Techmeme first spotted a story, not how long since the story was actually written, right?

“Old” in terms of how long it’s appeared on Techmeme. Though how old an article is according to its publication date is also a factor.

Q. Last month, Robert Scoble was explaining how Techmeme doesn’t pick up a source just because it links to an item on Techmeme. It picks up a source because other sources on Techmeme link to that source. That’s correct, right? To get into Techmeme, you need someone already in it to link to you — just linking to them doesn’t help.

That’s right. I think Robert recognized that that guy had a Technorati-type service in mind and believed merely linking out would trigger inclusion. That is, Technorati includes you in a URL search provided you link to that URL. Techmeme doesn’t work that way.

Q. But you did have a post with tips on getting on Techmeme in November where one of the ways was indeed said to be linking to you.

Yes, so if a post on your site includes a Techmeme permalink, as opposed to linking to an article that Techmeme links to, and my system notices a moderate amount of traffic through that link, and in addition determines your site is not spammy and the referral is real, your site stands a much better chance of appearing under the Techmeme “Discussion” for the article.

This is a good way for a news organization without RSS feeds to enable an article to appear on Techmeme. Without inbound links and without an RSS feed, many article URLs are undiscoverable by my system, except through this mechanism.

Q. When you launched way back, you had to have seed sites, correct? A group of sites you started out with. Do you still use that list?

Yes, though the list has been modified since then. Since September 2005, all of my sites have utilized a source discovery process in which the majority of monitored sources are found through yet not actually contained in the seeding set.

Q. How many sources do you monitor?

It’s in the low thousands. But on any given day, new sources are added and dropped so the total monitored over time is much larger.

I believe a good automated news site doesn’t really need to monitor more than a few thousand sources at a time.  Maybe even a few hundred will suffice.  Of course the key is the intelligence of the thing that sifts through those sources.

A good analogy for Techmeme is “automated blogger”. And there are indeed bloggers who largely work like filters or routers who churn through lots of stuff and only post the most interesting bits. Instapundit is a classic example. These people can only check a few hundred blogs daily, even the ones who work extremely efficiently.

Q. I’m exactly like that. I monitor like 100 to 200 feeds, to build our search headlines for the SearchCap newsletter each day. And if I keep coming across a blog being mentioned by other blogs with good stuff, I eventually add them to my feeds.

Right. Now Techmeme’s not as smart as you, but it can run faster, and there’s a lot of tedious stuff it can do that you would never want to do. So Techmeme is kind of like a blogger, but with strengths and weaknesses that go with the automation.

Q. Why do sources get dropped? Do they fail to post new material? Fail to keep being cited?

Fail to keep being cited. Every day Techmeme performs a bit of a reset, usually around 3am Eastern, where it doesn’t update for about an hour as it repeats the source discovery. So every day it tries to find the best few thousand sources. A blog can make the list one day but not the next.

[NOTE: I've seen this personally. My personal blog Daggle, for example. Things that I post there, I never assume they'll make Techmeme. But when I announced I was leaving Search Engine Watch, suddenly it got on and went to the top of the page for a bit. Links from elsewhere made it relevant for inclusion. Occasionally I see it make it for other things. But with Search Engine Land, we're far more likely to be in on a regular basis, since I gather we say things that enough other people in Techmeme find interesting enough to link to regularly.]

Q. Is spamming much of an issue for you? Since just linking to a story doesn’t get you on Techmeme automatically, seems like that should reduce a lot of it.

Exactly. I designed the algorithms to discover the most consistently useful news sources, and as a result spam blogs almost never come up. It certainly helps that reputable blogs almost never link to spam blogs.

Q. Do you spider everything, the full text of posts. Or do you just read whatever’s in a feed.

If my robot sees an indication that there’s additional text on the article page, it’ll spider that page. I need to make it a little smarter so that it succeeds in this more reliably, but I’m most of the way there even now.

Q. Do you store the full text over time?

Yes. Though not readily retrievable by all parts of my system, the data is archived.

Q. Do you depend on feeds to know who is authoring a story?

I utilize both the feed and the main page. I found early on you need to visit both to accumulate all this metadata.

Q. How’s the sponsored posts program going since it launched last September?

I think it’s going pretty well. The obvious test being that sponsors are renewing. I’m even sold out for the next few months. Now the approach I’ve taken tends to limit who can sponsor Techmeme, so I don’t have a network of thousands of potential advertisers like Google, but fortunately there are enough sponsors out there to fill Techmeme’s inventory.

Q. How the Techmeme River going? Lots of use?

Not lots, though I wasn’t expecting lots. The river was intended just to cover some unusual uses cases. For example, when you need to scan everything that’s happened over the past few days, or you want to search through recent Techmeme posts for certain authors or title keywords.

Q. How about a way to see the top things on Techmeme, the most popular items rather than the river, that just lists them by date? When I was at Popurls, I was surprised not to see the top stories from Techmeme shown right up there with the top picks from Digg, Delicious, Reddit and others. Perhaps a popular list would help?

What you want is basically the Techmeme RSS feed. It includes the top 20 to 30 items of the day. Perhaps I could create starting page based only on these posts, but others have effectively done so already. For example, Original Signal includes Techmeme.

Q. Is Techmeme too elite with its sources?

Any selection of finite length is necessarily elitist. Now people often protest about Techmeme’s elitism, but these people are typically bloggers who don’t show up on Techmeme frequently. I don’t believe I’ve had anyone ask me make it less elitist that doesn’t blog! Yet my readers are largely non-bloggers.

For better or for worse, well-read bloggers tend to have better access to interesting news, and also tend to exercise the talent that helped establish them in the first place. I’m rather unapologetic that there are lots of less established writers who will never show up on Techmeme.

Q. Do you hand remove or add sources?

If I believe a site is essential enough, I’ll add it to the seed set, and I do that occasionally for Techmeme.

Q. Does it make any sense for people to ask you for inclusion or does it make more sense for them to get the attention of A-listers already in Techmeme?

Getting the attention of bloggers has probably worked for many more people than has asking me. Trackbacks, emails, and comments can be very effective and require much less time than the actual reporting and writing. And appealing to “A-listers” isn’t even necessary. The “B” and “C” list also wield considerable power in the Techmeme ecosystem.

Q. How are the other sites doing? [WeSmirch for gossip, Memeorandum for politics, Ballbug for baseball news -- all use the same Techmeme technology]. Do you have more planned?

The trending looks good on WeSmirch, the celebrity gossip site, which apparently has decent word of mouth. Ballbug, the baseball site is clearly the worst of the bunch trafficwise. The political site, memeorandum, is doing OK but could do better.

I think I want to further develop the technologies that improve the existing sites before introducing even more sites. Although 10 minutes of actual labor can lead to a new vertical, launching something truly compelling will depend on new technologies. Fortunately, these are the same technologies that will improve my existing sites.

Q. Ever been on Digg?

Techmeme has never made the front page or any category pages of Digg. I believe a few links have been submitted, but I suppose they were of little interest to the Digg community!

Q. Why not sell up or work for a larger company?

I’m not sure there’s a larger company that sees eye to eye with me about where all this is headed or where the value is and isn’t, to be perfectly vague. Probably the biggest reason though: I nap after lunch, unconditionally, and most big companies can’t accommodate this need nearly as well as my own.

One of my resolutions in starting Search Engine Land was to spend more time doing Q&As with people. There’s no better way to ensure I do that than to start a column. So I’m happy to announce that Q&A Land becomes the second column offered by Search Engine Land.

Unlike Link Week, which started this week and runs on Mondays, Q&A Land is a “floating” column. It will happen once per week, but the actual day might change.

I won’t be doing Q&A Land alone. Chris Sherman will also be writing for it, and from time-to-time we may have other guest authors. And yes, we’ll have a nifty logo coming, as we already have for Link Week.

The first official Q&A Land column is now out, Q&A With Gabe Rivera, Creator Of Techmeme. I say official because I’ve also now classified my Q&A With Jimmy Wales On Search Wikia article from the end of December as part of Q&A Land as well. Check them both out!

News came out earlier this week that Wikipedia founder Jimmy Wales had a new project in mind, to build a community-driven "Google-killer" search engine. I’ve just finished talking with Jimmy about his plans. Here’s a rundown on his vision and what may come as his Search Wikia project grows over the course of the next year or two.

Note that in the Q&A, I’ve had to recreate my questions as best I remember asking them. I was focused more on getting down Jimmy’s responses.

Q. Since the news emerged, there’s been some confusion about Amazon and Wikipedia in relation to Search Wikia project. What’s the situation?

We recently completed a funding round with Amazon [for Wikia], but other than that, they don’t have anything to do with the search project. [The project] is a Wikia project [the for-profit company that Wales is chairman of], not a Wikipedia project [the separate community-driven encyclopedia he co-founded].

Q. Was the search project formally announced, or did the Search Wikia site come online as a result of The Times article discussing it.

It was a combination of them both. I’ve been working on this for a long time. We didn’t actually intend to announce per se just yet, but me and my big mouth, the reporter asked me if I ever thought about search.

Q. It’s been said the search engine would launch in the first quarter of 2007. That’s fast. Is that really just when you expect active development work to begin?

During Q1, we’re going to set up a project to get developers involved with building the site, writing the code and getting the search engine going. We’re going to rely initially with Nutch and Lucene [related open-source search software that's been developed over the past few years].

We’ll start from scratch on how to apply the Wikipedia principles to keep it as simple as possible and move forward.

It’s just the development starting. We’re not producing a Google killing search engine in three months. I only wish I were that good of a programmer.

We’ll have some servers open, some development, maybe a pre-pre-alpha demo site up. We’d really anticipate it would be a year or two until we’re able to launch a viable search engine.

Q. How do you see this improving on what’s out there?

There are a lot of things that we’ve learned in the wiki world on how to get communities involved and engaged to build trusted networks in communities.

A lot of the people who have tried to do this in the past have stumbled not on technical issues but on community issues … dmoz [The Open Directory] was too closed … that was their response because of the pressure of spammers … others have thought in terms of ranking algorithms. That’s not the right approach. The right approach allows for open dialog and debate and discussion.

Q. How do you envision the community participating? Will they be selecting sites? Will this leverage material in Wikipedia? Will they rate sites?

This will be completely independent of Wikipedia.

Exactly how people can be involved is not yet certain. If I had to speculate about it, I would say it’s several of those things, not just community involved with rating URLs but also community rating for whole web sites, what to include or not to include and also the whole algorithm … That’s a human type process that we can empower people to guide the spider

Q. Do you see humans reviewing the most popular queries, perhaps picking the right answers to come up?

Part of it might be a human review of queries. For the narrow subset of the really popular queries, I think it’s important to apply humans …. if someone types Ford Motor Company, there is a correct answer for that. There’s no reason to beat our brains out to train our algorithm to do that.

Q. Search engines have actually gotten much better over time with these type of navigational requests. You don’t need humans so much to make sure the right answer shows up.

Those kinds are not too difficult. The harder one if you type ford, did you mean President Ford or do you mean the Ford Motor Company? That’s the type of thing where human disambiguation pages like we have at Wikipedia are helpful.

Q. Search engines already do a lot of this type of stuff. Ask has its Zoom suggestions, others have clusterings or related searches. Do you imagine people being forced to make a query refinement choice before they actually get search results?

If you type ford, you should get some disambiguation terms that humans have collected, then some search results….this is one of the places where I think human intelligence is most important

[NOTE: For more on query refinement, see some of my past posts such as Robert Scoble Wants What We Had -- Better Query Refinement. So Do I!,  Hello Natural Language Search, My Old Over-Hyped Search Friend and Why Search Sucks & You Won't Fix It The Way You Think. The first link in particular discusses how Microsoft used to have disambiguation created by editors very similar to what Wales hopes to recreate. Sadly, it was killed in the quest to chase Google on the algorithmic front.]

Q. Are you planning to crawl the entire web, billions and billions of pages? Or will you go after a subset of important ones?

The number of pages is yet to be determined. Obviously we won’t be doing that initially [gathering everything], but we’ll invest in the hardware. Not to belittle the investment required to do a full crawl of the web on a regular basis, but I think it’s a fairly commoditized.

Q. Crawling is one thing. Serving up millions of queries per day is an entire other issue. Wikipedia handles a lot of traffic, but not at a Google scale. How’s it going with that?

The traffic’s not too bad. Servers are getting more and more powerful. Bandwidth is getting cheaper. It’s all pretty much off the shelf. It’s pretty efficient.

Q. Will you be selling ads, and if so, how will that work?

There are no immediate plan to sell ads, so for now we’re not too focused on that. If we don’t build something useful, selling ads on it is sort of a moot point.

Q. Why do this at all? What do you see wrong with search?

For certain types of searches, search engines are very good. But I still see major failures, where they aren’t delivering useful results. I think at a deeper almost political level, I think it’s important that we as a global society have some transparency in search. What are the algorithms involved? What are the reasons why one site comes up over another one. [Wales also raised the issue of how ads might influence regular listings, perhaps search engines trying to keep commercial sites out of the free listings to make money. From there, he went on....] Those types of incentives are problematic in search. The only solution I know to that is to be transparent

Q. How are you going to keep the community from being gamed. Wikipedia is very good at keeping out spam, but it’s not perfect. And despite its size, it’s dealing with far fewer topics than unique searches that will happen on any particular day. How do you police all those searches?

You have to recognize the difference between the way community is often used on the internet, which is short hand for millions of people clicking on some stuff as compared to community in the wiki world, which is people who actually know each other.

It’s one thing to say if you have millions of spammers out there trying to game and trick an algorithm …. but it’s not the number of queries. it’s the web sites themselves. A lot of numbers are thrown about for sites on the web, but the number of legitimate pages that are not coming from affiliate sites and spammers is a much more finite number. It’s much easier for a community to ban the bad stuff.

Q. But what if someone gets into a "good" domain. We’ve had cases where bad content gets shoved into "trusted" sites or even places like university sites. Do you ban those entire domains? How do they get back in?

At Wikipedia, we’d have a big discussion. [Wales then explained that people might realize a domain had done something accidentally wrong or without thinking about spam issues and so might be allowed back in.]

Q. You probably already search a lot, probably mostly with Google. Is it not finding what you want already most of the time, without a flood of spam or crud in your way?

Usually I’m looking for pages on Wikipedia, so they do a good job with that. It depends on the types of searches you are doing. If you’re doing a factual search, then Wikipedia [in the results] would be good. In other areas, I think there’s a strong commercial incentive. Why is it bad if I search for tampa hotels?

[NOTE: I then did this search on Google, which we discussed. I noted I saw plenty of good hotels listed, and that if I clicked through to the local search results, I got an even better experience of hotels listed.

Wales replied that he's often after reviews of hotels, not the hotels themselves. That took me back to the original results, where I pointed out the top listing was from TripAdvisor, exactly the type of review site he mentioned liking -- and that I often found them listed on these types of queries.

I also noted that Google even offers refinement categories at the top of the page similar to the disambiguation he wanted, with lodging guides as one of the categories. Unfortunately for Google, I didn't find that the results from that refinement did a good job bringing back trusted hotel guides]

Q. Back to transparency. People keep saying they want more of this. But can you name some exact examples of what you want to see? Do you want Google to say that using a term in bold text adds X percent of a score to the ranking criteria? And if you do that, don’t you think spammers will just abuse the recipe that’s been published?

If your search relies on some secret factors that you hope people won’t discover, you haven’t really come up with a good solution the problem.

Q. Microsoft has spent millions of dollars and years now of effort to try and be a Google killer and haven’t made it. You’re coming into this fresh with fewer resources and no real prior experience. Can you really do it?

I have no idea. I only do whatever sounds like it is fun.

Q. What type of funding do you have behind this?

Wikia’s initial round was 4 million from a variety of angels, then there was second round from Amazon, but the amount wasn’t announced.

Closing Comments

When I first heard of the plans, I was pretty dubious the project would have much success. For one thing, the idea of the "open source" search engine to take on the world and provide more transparency is old news. Consider this from back when Nutch first came out, out of New Scientist in 2003:

The project "is about providing free technology that should not be controlled by private, commercial, secretive organisations," says Doug Cuttings, veteran web search engineer, and a Nutch founder.

Three years on, nothing really changed despite the reasoning behind such a project being the same. And this was despite Nutch having some big names behind it.

In 2004, Nutch got another round of attention in an ACM article looking at how it works. My comment at that time was:

Interesting read especially for the efforts that are involved to defeat spam. The argument is that though Nutch is open, revealing secrets won’t hurt because spammers will batter down any defenses, no matter how tightly protected. OK, so what will stop spam? Nutch hopes that an open, public discussion may reveal new methods. Perhaps. But the real test will only come if Nutch is deployed by a major, highly-trafficked site. Spammers aren’t going to bother trying the defenses of other places. It’s not worth the time. That’s also a positive for those considering Nutch. If you operate a small, vertical site or just want Nutch to be used on your own content, then spam concerns are much less an issue.

The spam test simply hasn’t happened with Nutch. And every new search engine project I’ve looked at coming in over the years completely underestimates the spam problem they face. When I looked at the Search Wikia site, comments like this almost seemed laughable:

search active for spammer sites

• trying to simulate user-typos (ie. "yaoho.com" rather than "yahoo.com"); see also: Microsoft’s URL Tracer
   
• blacklist domains, where spammails are linking to; create actively honeypods to get spam; use a pattern like <domain-where-we-have-registered>@myhoneypod.com to identify the spam networks; shell the common user get the possibility to register such a mail-adress?

Seek out the spam sites? Hey, don’t worry — if you’re popular, they’ll find you fast enough. And as you blacklist one, two more throwaway domains will show up in their place.

I also tend to think Wales is completely underestimating how crawling a big chunk of the web, keeping those pages fresh, ranking them quickly to provide answers and doing so for millions each day isn’t an off-the-shelf commodity.

Still, I find myself oddly hopeful. I don’t think a Google killer will emerge, but perhaps some new ways of a community to be involved with search will come out of it. I wouldn’t have thought Wikipedia would work. Certainly it’s flawed, but it’s also an incredible resource. Maybe something useful will come from the Search Wikia project.

At the very least, I’ve long wanted humans to be back in the role of reviewing queries and actually looking to see if they make sense, rather than so much reliance on algorithms. Maybe the mere concept of the Search Wikia project will encourage the major search engines to do more in this area.

Postscript: Originally I had Wales listed as cofounder of Wikipedia, but he got in touch saying he was the founder. I’d noted that Wikipedia itself lists him as founding (well, creating) it with Larry Sanger. Is Wikipedia incorrect on this, I asked? “Yes, it is wrong,” he emailed. Sanger posts his own views on the origins here.