Here at Search Engine Land we regularly hear from people who have created “radical,” innovative,” “next generation” search technology, promising to “fix” the “broken” search we all allegedly pitifully struggle with today. In virtually every instance that I can remember, these promises over-hype and under-deliver, rarely offering something that becomes part of my regular search arsenal.
But the bizarrely-named DuckDuckGo is different, for a number of reasons. First and foremost, it’s actually a very good search engine—and it truly can do useful things. Second, the creator of DuckDuckGo has a nearly maniacal obsession with privacy, and has built very powerful tools into the search engine that go to great lengths to keep your queries (and your identity) anonymous.
DuckDuckGo is also remarkable in that it’s the creation of a single individual, serial entrepreneur and angel investor Gabriel Weinberg. And rather than hyping his creation as so many others do, Weinberg has kept a comparatively low profile, doing little to promote the search engine since launching it in September 2008. Despite this, DuckDuckGo has developed quite a following, with 5 million search queries this month.
Since the first of the year, Weinberg has been more aggressively marketing the search engine with a campaign aimed squarely at Google, with the slogan “Google tracks you. We don’t.” For those of you who aren’t aware of the various ways that Google collects and shares information about your searching behavior, you might find Weinberg’s claims disturbing—even shocking. More on that, and how DuckDuckGo differs from Google, below.
The Anatomy Of A New Search Engine
After founding and then selling NamesDatabase to United Online in 2006 for about $10 million in cash, Weinberg started work on DuckDuckGo as a weekend hobby. Why a search engine? “It’s the technical skills you get to mess around with,” said Weinberg. “It involved databases, text mining, web crawling—all things I’d done previously.”
Weinberg’s original idea was to search on a mashup of Wikipedia and Delicio.us, using the high quality external links from each service as results. “What would happen? I messed around with it and liked the results for the fat head of the search spectrum,” said Weinberg. Next step: development of a web crawler and inclusion of a number of other high-quality “curated” sources.
From the outset, DuckDuckGo was designed to be fast—a reasonable goal these days with other search engines typically returning results in a second or less on broadband connections.
DuckDuckGo uses its own crawler (DuckDuckBot) to compile an index of the web. It also uses data it gets from more than 100 crowd-sourced sites, Yahoo! BOSS, embed.ly, WolframAlpha, EntireWeb and Bing, making it something of a hybrid metasearch engine—hybrid, because while it uses many data sources, it also does some post-processing of results to filter out parked pages used for nothing but ads and other types of non-useful content. So rather than simple blended results you get from most metasearch engines, DuckDuckGo tries to pull together the best it can find from multiple sources.
Eliminating spam from results was also a top priority. Weinberg says that DuckduckGo’s spam list currently numbers more than 70 million domains, and is growing.
Want a really detailed overview of how DuckDuckGo works? See DuckDuckGo architecture (warning: serious tech knowledge required to fully absorb this article).
Seriously Cool Features
As a Google or Bing-like general-purpose search engine, DuckDuckGo more than holds its own in terms of relevance. And its algorithms differ enough to provide a genuinely different view or “opinion” of quality content on the web. Beyond this, though, DuckDuckGo has some very powerful features that make it a very compelling choice for many types of searches.
Browser Icons Next To Search Results. You know those little icons that display on tabs when you’re looking at a web page? Sometimes called favicons, they’re unique to each website—typically some kind of stylized logo (look at the tab on this page and you’ll see Search Engine Land’s “gear” icon. This is a minor feature, obviously, but if you’re heavily right-brained and use images for recall like I do, it’s a great way to quickly identify a source at a glance.
Zero-click Info. Displayed above traditional search results, zero-click info is sort of a combination of instant answers, best page preview, quick facts and other snippets of information that DuckDuckGo has determined to probably be most relevant to your query. Both Google and Bing have similar features, but are typically presented in a consistent (therefore constrained) format. DuckDuckGo’s zero-click info appears in the format that makes the most sense for a particular query.
Categorized Topic Lists. Sites appearing in zero-click info boxes are categorized. Click the category link and you’ll get an alphabetized list of all DuckDuckGo sites in that category (here’s the internet search engines list). This is a great tool for getting a quick but comprehensive overview of, say, all companies operating in a particular industry. Many famous people have their own category lists as well. Web directories have had these categorized lists for years, but they’ve always suffered from web rot as listings don’t change to keep up.
Easy Site Search. More often than not, site search tools fail to work the way we expect. DuckDuckGo offers direct site search of “hundreds” of sites. You can access these direct search sites by using a dropdown menu next to the search box, or by using the “bang” command (an exclamation point) next to the name of a site, company name, keyword, etc. Check out this full list of DuckDuckGo bang keywords by category.
Tools. DuckDuckGo offers a number of free tools, including official and unofficial apps and add-ons that extend the functionality of the search engine. It’s also the default search tool for a number of lesser known browsers, if you want to try something other than Firefox, Internet Explorer, Chrome, Safari, etc.
Even More Goodies. DuckDuckGo can do many other things, like performing calculations or looking up specialized information. These are particularly useful if you’re coding or doing website related work. Check out this list of DuckDuckGo goodies.
Privacy And Anti-Tracking
Google And Privacy
Google has long been an advocate of both privacy and transparency in terms of how it handles personal information. It has an extensive Privacy Center that offers both specifics about it’s general policies, and Privacy tools that let you make choices about many types of personal information Google collects about you. Google has five privacy principles that help guide decisions about all of the company’s products and services:
- Use information to provide our users with valuable products and services.
- Develop products that reflect strong privacy standards and practices.
- Make the collection of personal information transparent.
- Give users meaningful choices to protect their privacy.
- Be a responsible steward of the information we hold.
Google has historically fought hard to protect user privacy in the face of external demands for information. A notable example was Google’s response to a U.S. Department Of Justice subpoena requesting all URLs in Google’s index and all queries to Google over a two-month period as part of an investigation into child porn (for a particularly eloquent and impassioned defense of Google’s practices, see Matt Cutts’ declaration in support of Google’s case). In general, Google deserves major kudos for its approach to privacy, transparency, and empowering users to make choices about how their personal information is used.
Despite all this, DuckDuckGo maintains that Google isn’t totally walking the talk when it comes to privacy. Let’s look at why.
DuckDuckGo Vs. Google On Privacy
No Search Leakage. DDG: “At other search engines, when you do a search and then click on a link, your search terms are sent to that site you clicked on (in the HTTP referrer header). We call this sharing of personal information ‘search leakage.’ In addition, when you visit any site, your computer automatically sends information about it to that site (including your User agent and IP address). This information can often be used to identify you directly.” (For more details, see Why You Should Care – Search Leakage.)
Google and others challenged Weinberg’s assertions as fostering fear, uncertainty and doubt. Weinberg responded on his blog with Search leakage is not FUD. Google et al., please fix it. This in turn prompted a response from Google’s Matt Cutts on Hacker News followed by a lively discussion. Cutts points out that Google now offers secure search via https://www.google.com. True—but how many the people doing over 11 billion searches per month on Google either know about this privacy feature, or take the time to append that simple “s” to the URL when searching?
No Search History. DuckDuckGo doesn’t save your search history. Google does—but only if you sign in with a Google account, and you can pause or disable history at any time by signing in to your Google Dashboard. DuckDuckGo: “With only the timestamp and computer information, your searches can often be traced directly to you. With the additional account information, they are associated directly with you.”
So… Who’s Taking The Right Approach?
As I wrote above, I really like some of the features offered by DuckDuckGo, and would likely use it without its privacy safeguards. I also really like a lot of Google features—including search history that personalizes search results—and I trust Google to safeguard my personal information to the best of its ability. Some may consider this naive, but I think there’s a tradeoff here between privacy and utility that requires a personal decision from all of us.
It’s helpful to consider what “privacy” means in an offline context. For example, every time you make a purchase with a credit card, that information is not only sent back to the card issuer, but is also often shared for marketing or other purposes. If you walk or drive around any major city in the world you’re videotaped by multitudes of CCTVs. Any public record (including “official” records as well as social media or other web content that you’ve not protected) can also be easily shared. So the concept of privacy is in the eye of the beholder, and again, requires a personal decision from each of us about whether we want to use certain search engines or other services, either online or off.
So I’m not taking sides on this one. I think both Google and DuckDuckGo are operating in the best interests of their users, and privacy decisions are best left up to each individual.
What’s With The “Quacky” Name?
Apart from the domain name being easy to register due to non-existent demand from domainers, Weinberg claims that he just liked the name. It is derived from Duck Duck Goose, a children’s game. For those of you who prefer shorter names, you can also access DuckDuckGo through the http://dukgo.com URL.
“The long-term goal of DuckDuckGo is to get as many people as possible to want to use it as their primary search engine because of a great search user experience,” Weinberg wrote on his blog. He told me that he has no plans to cash-in over the short term, and is fully committed to continuing to improve DuckDuckGo for the foreseeable future. He would also like to offer search advertising opportunities, but at this point has found no ad networks that would work within the strict privacy guidelines he’s developed.
Weinberg is aware of search engines that have launched with a lot of hype and hoopla, and is careful not to make the claim that DuckDuckGo is a “Google killer.” And he has a sense of humor about over-hyped search startups of the past.
Remember Cuil, the search engine launched in 2008 just a few months before DuckDuckGo’s debut? The one that was widely touted by the media, struggled with numerous technical issues, and then quietly shut down last September?
Weinberg poked fun at all of the Cuil hype by registering the domain names Cuiler.com and Cuilest.com, mimicking Cuil’s interface as a front-end to DuckDuckGo. Those links still work—give them a try if you want a good chuckle.
- DuckDuckGo Challenges Google On Privacy With DontTrack.us
- Class Action Suit Targets Google, Seeks Elimination Of Search “Referrer” Leakage
- The Death Of Web Analytics? An Ode To The Threatened Referrer
- Google Anonymizing Search Records To Protect Privacy
- Anonymizing Google’s Server Log Data — How’s It Going?