Since Wolfram|Alpha launched in 2009, I’ve often wondered if the company would allow users to use their extremely powerful computing infrastructure to analyze their own data along with data sets available on the open web.

Well, they have and that’s what Wolfram|Alpha Pro is all about. As I wrote a few week’s ago, I’m a frequent W|A user and long-time admirer of the company and it’s founder, Stephen Wolfram. I think they’ve scored again.

The service has an introductory rate of $4.99/month with a reduced rate of $2.99/month for students. The company offers pricing for the enterprise. So, what does this initial launch offer?

Wolfram|Alpha Pro: Key Features

• Ability to Analyze Your Own Datasets Input of upload a data set and let Wolfram’s technology automatically recognize and analyze the data. For example, a timeline of your email, your credit card expenses, or a data set obtained from Data.gov.

• Data in more than 60 file formats can be uploaded and analyzed. From an XLS spreadsheet to an HTML document to the audio in a WAV file. Image files (in many formats) can also be analyzed. As you would expect, many science and math formats are also included.

• Access to extended keyboards like the one available with the Wolfram|Alpha mobile apps.

• Visualized and interactive results using the CDF (computational data format) that Wolfram launched last July.

What’s not available today is the ability to publish and compare, comment on or share results. According to Stephen Wolfram, these options are forthcoming.

I’ll can say now that while some of what Wolfram|Alpha Pro has to offer might not be for everybody today it very likely will be in the future as the amount of data available to analyze (personal data and publicly available datasets) continues to skyrocket.

In other words, Wolfram|Alpha pro is getting in on the ground floor of personal data analysis with a powerful but at the same time very easy to use tool that will appeal to everyone as data analysis moves out of the lab and office and into just about everyplace else.

Postscript by Barry Schwartz: On February 8th, Stephen Wolfram posted more details about this Pro offering on their blog.

Should be a fun day. See our previous post for the schedule. Live webcast will happen off the Big Think home page. Buckle up, and my live blogging begins at 10am Pacific.

And here we go. Victoria Brown is welcoming us. She’s from Big Think, talking about how she talked with Stefan Weitz from Bing some time ago about doing this session.

Now Vivek Wadhwa is up. Talking about how search hasn’t really changed — you enter words and get links. The main change is, he says, the web has been overrun by spam. Does no one remember the spam glory days of Infoseek and AltaVista? I know some old fart SEOs do.

Vivek is talking about now wanting his computer to know about the type of restaurants he wants to go to, who his friends are. He’s hoping today we hear good idea and hear good news about how the web is going to change.

Now we get Peter Thiel doing a keynote. There are all types of interesting tech that can be built, woah, he just said Powerset that he invested in became Bing. Not. So. So not so. Anyway, beyond tech, let’s talk the cash in tech. So far, Microsoft still doesn’t seem to be making money. He estimates you need 35% marketshare to get to break even. Hmm. Hmm. I don’t know. Not an economics guy. But I’m pretty sure we had search engines profitable with lower share. Especially if you don’t mind shoving out ads to an unknowing audience.

Until fixed costs are solved, you have a monopoly. As investor, he’s only looking at companies that seem to have dealt with that (I think he said). As for Powerset, it ran into the fixed cost problems. They badly estimated that. Not to mention that Powerset wasn’t a general purpose search engine and was, to my understanding, a hell of a lot processor intensive to generate result that weren’t as useful for most searchers than regular search engines.

Long talk about economics, concluded by saying only Microsoft has the capital costs to compete with Google. Sorry, Blekko.

Now we’re getting the lead-up to the “Who Will Win The Spam Wars” panel, with Matt Cutts from Google, Harry Shum from Bing and Rich Skrenta from Blekko. This should be fun.

So Matt, what about the mess, Vivek asks, meaning spam. Matt’s skipping this and instead diving into the story about the Google says Bing is copying them thing.

So Matt asks Harry, what about this whole thing we found? Harry says interesting that Matt talked about all the improvements that Bing has, and you can’t discount all the work that Bing does.

But in his view, he thinks they’ve just discovered a new form of spam or click fraud. He wishes customers could take that to them, so they’d have time to study it. Hmm, sounds like he’s saying Google was doing something fraudulant.

Matt’s referring to a few outlier examples, he says. Bing looks at a lot of different signals. It’s not like we copy anything. We learn from the customers … from what type of queries they type.

Matt’s saying well no, those signals Bing uses don’t seem to just be showing up in synthetic tests.

Harry says you have to be careful. We learn from our customers. “Did you mean that Google owns the data, because the person uses the Google search engine.”

[You all getting this now? Bing probably won't say they copy Google, because it seems to argue that it's merely monitoring uses. Which, is also true, though the end result might be the same]. Matt went off on the Suggested Sites feature and I think says it’s not really clear what data it takes. Harry says Google well Google does the same. Matt says oh no we don’t and snaps his fingers. Except that was in his mind. He actually says categorically we don’t.

And Rich is either thinking whew I’m not being yelled out or when do I get to talk. Oh, he just got called on.

===

The web is full of garbage, he says. Now he’s gone into some numbers, which I lost track of, sorry, live blogging on tape is hard. That’s a Real Genius reference.

Basically, there’s all this stuff dumped on the web each hour, and it’s all designed to deceive search engines, because there’s value to being in those search engines.

Matt? He’s completely right. Likes they have SpamClock.com with 1 million pages per hour, and Matt’s first thought was only a million? ====

Vivek asks about Blekko’s banning of major spam sites. Matt says we prefer to have an algorithmic approach than manual action, on the whole.

Harry? Google is responsible for so much of the spam we see, why they appear (think he means because of AdSesne). Yes, why did they do that — Google ads. It’s easy to figure out the percentage of AdSense sites that are spam. You guys understand more of what going on and have a responsbility to share so we can tackle this problem jointly. Wow, we’re totally gloves off now.

Matt, many don’t realize if the spam team finds and ejects someone that kicks them out of AdSense. and if there were no AdSense, we’d still have spam.

Vivek, but Matt, what’s your incentive to clean all this up. Matt, we’ve always had the philosophy to do the best for users. Wouldn’t do pop-ups, for example. ===

Vivek, but how about content farms, and places like Demand help you earn. Matt comes back to saying you need an algorithm. Rich gets asked about algos, but didn’t catch all of what he said, sorry. Now Harry’s talking about Blekko says watches what they are doing. Well, technically watches what Blekko users are doing. C’mon, give me a laugh

Harry goes on, we really need a combination of manual intervention and algorithms. And one thing that needs to be looked at is the notion of authorship, do authors really have the authority to do well.

Vivek, what’s the difference between the three of you? How are you better than each other.

===

Shit, now I’m getting called out. Oh, it’s OK, it’s about one of my earlier article that Harry’s talking about, where I said we really need better metrics that everyone agrees on about measuring relevancy. Harry’s going to talk to me after the panel about this. And I shouldn’t worry about that big piece of wood he’s holding behind his back :)

Matt’s saying Google has a bunch of evaluators and that’s how they think about this stuff. They work hard to be fast. Spam’s an issue, but I think he’s saying it’s not as bad as some other issues.

====

Harry says they do have internal metrics, too. But he says we should all come together as an industry to come up with stats. Google as the leader has responsibility to do this.

Rich is talking! It’s not the algorithm and depending on that. It’s editorial decisions too. And I didn’t catch part of what he said. I’m really, really sorry. Have I mentioned having no sleep last night?

Harry’s talking again, and he really wants to get some defined numbers. He’s waiting for Matt to sing kumbaya on this with him.

Matt’s saying that you can try to prerank a bunch of queries and do other type of tests to figure out if things are working or not.

Rich is back, talking about how his slashtags allow for curating the best content, as well as the bad content — all assembled by humans, so why not leverage both of them, killing the spam.

Matt says he’s got a spam blocking tool that’s in testing to let individual users block individual sites they don’t like. This might come.

====

Where’s the web going. Didn’t catch some of what Matt says, but he says Google is being very transparent.

Harry says he agrees with Matt, search quality is everything. So far collectively, there’s a lot of good work. But human judgement is hard when library on West Coast can means something different than on East Coast. The goal is to go one step behind and figure out what’s in people’s minds and help them actually achieve it.

Rich, our vision is a shared curated web that is free of spam. Honest, that all came out, nice and clean, tweetable. Someone at Yahoo, hire Rich.

You have to use the Wikipedia model and crowd sourcing. The only way to clean up the web is to bring humans back.

===

Matt, love the idea there might be 20 more Blekkos. It proves with a small team, you could do amazing stuff.

Harry, what about dealing with costs, as Thiel talked about earlier. It’s just the kind of investment you have to do, cutting cost is one of his top things.

Rich, how do you do it? We’ve got $5 million in hardware, and you can do it. With $25 million overall, you can have a good web search engine. There should be more. We only have two engines in the market, so we should have a search. With search so valuable, why isn’t there more investment.

And that’s it. The three have gone off behind a curtain to start htting each other in public.

====

Jaron Lanier is up. I liked him at SXSW last year. Just saying. Talks about how Esther Dyson who is here once suggested a cost to email, but we want things friction free — but as a result, we have spam.

Now Jaron says he knows nothing about the whole Google copying Bing thing. He doesn’t think it was something willful, however. It’s just that as Bing looks at what people do, Google’s content leaks out. That’s kind of reasonable.

Going on, he says it’s kind of like with YouTube saying to the content producer that you need to produce it. Not to be partisan, it does seem to him that Google’s complaining about the same system that it benefits from.

Jaron thinks you need to have friction, as Esther has suggested. Said some things I didn’t catch, sorry, needed to multitask poorly for a second. And Jaron says he had slides that aren’t appearing — so see, that would have helped me, your poor live blogger.

But we need to move to a prior organization of data, things like Facebook where you have an enforced identity.

Jaron says he’d happily pay 1 penny per search, if it was easy. ====

Esther is up is now, talking about how Bill Gates told her at dinner that the future of search is about verbs.

In the past, search was about nouns. Hey, I like this I have a whole Google is a noun, Foursquare is a place, Facebook is a person, they’re all nouns thing. But no, it doesn’t keep doing in that direction.

The new form of search is about verbs. Not just the location but how do I get there.

So is the future of search an entire new market. If Google gets ITA, Bing with Bing Travel … can make the argument that as the web is going app like, search engines are a set of apps.

====

OK, Di-Ann Eisnor from Waze is up. She was sitting next to me a minute ago and is very nice. They gather paths that people make, as they travel. They provide real time GPS info based on crowdsourcing. Where should I go. They’ll give the ETA right now. they watch for tweets, things like hazards.

Enter your destination, enter your ETA and they give you options. The have prizes in places where they don’t have info, to get you to go there and get points. Hey, Foursquare meets GPS.

Lots of stats and demo of the service.

===

Here’s Blaise Aguera y Arcas, a shining star at Bing. Talking about moible, old view was a dumbed down web, 1.6 versus double that searches on desktop. None of this is true now — mobile is much more desktop like.

Average voice query length is 3.5, almost like on a PC. Plus you have phone sensors, like camera, GPS that can help.

Lots more stats and observations.

Search 1.0, looking for nouns, browser-based, read-only, lots of things — I’ll add a screenshot in a bit. But Search 2.0 breaks all these things.

Search 1.0:

Search 2.0:

===

I’ve had to go out for a bit, so I’ve missed some of the panels. Jumping back into the individual presentations.

Gary Small, Director of UCLA’s Center on Aging, is up. He’s talking about that whole is Google making us stupid thing. His team tried to discover how search is affecting our brains. So he needed poeple who’d never been online as a baseline – these were pretty much older people. Then they matched them to similar older people (62ish) who were net savvy’

Put them in MRI machines, showed them search pages to review. The “net naive” had same images when dealing with search or text. But with savvy people, your brain got active twice as much when doing searches. Not twice as smart, as some reported, just more activity.

Gord Hotchkiss actually did a big article about this for Search Engine Land, and how brains can be trained in relation to search. See it here Is Google Rewiring Our Brains?

So is tech weaking our memory? We have to pick and choose what we store in our brains.

Surgeons who play video games make fewer errors.

In future, he predicts we’ll have headbands or earpieces, you’ll think to your computer, which in turn might talk to a friend’s computer and then into their heads.

Computer mindreaders. CMU is developing experiments where computers read thougths. Really? Wow, I have to look that up.

In the end, not hurting our memory but changing how we use it — and an opportunity to shape how we use it.’

====

Ilya Segalovich, cofounder and CTO of Yandex, who says he was asked to talk about why Yandex is so popular in Russia. Yes, Russia is one of the few places where Google isn’t number one.

In 1991, there was no good coverage of Russian web pages. They’ve grown over time, closing on 70% share of the market. Covers important vertical search markets, too.

Search quality — think its important to have internal and external metrics (there’s a company that tries to measure externally). When made new algorithm a year ago, had sharp rise of marketshare.

Strong focus on local audience. 77 regions in Russia; 5 countries that use Cyrillic alphabet.

What could Google steal from this, is a question. Good shopping, bilingual search are some things he tosses out.

===

And now, my favorite TV & movie search engine, Clicker.com. Paul Wehrley, cofounder and COO, is up.

They only list where legal content is located on the web; they know seasons, high quality meta data, tons more good stuff.

Why start it? A few years ago, there was enough core data online that could be attacked. Knew they needed to come up with a structured way to get to the content.

For Clicker, relevancy means getting rid of “noisy” content and links to illegal content.

Among the solutions is user data, watching what people do — they have a new partnership with Facebook to personalize results (it’s kind of neat).

===

Q&A panel now, and I’ve just been listening, for once. But an interesting nugget from Segalovich who wants to see search history portable. And by history, he means all the things you’ve clicked on and searched on should be transferable from one search engine to another.

That brings him back to today’s Google-Bing dispute. Looking at it this way, he says it’s not Google’s data that’s somehow being taken but the user’s data that they want to take.

I thought Google did allow for search history to be exported. However, looking at Google’s Data Liberation Front site, I don’t see search listed.

And with that, the event concludes.

As sites and pages multiplied exponentially Google became an increasingly necessary tool. We’re all familiar with the story. Now Google controls a majority of search traffic in most countries around the world. It has become a seemingly unstoppable force.

The influence of search (paid and organic) has been so powerful that billions of dollars have changed hands and established media companies have been all-but-toppled by their failures to recognize and exploit search effectively. Newspapers in particular fall into the latter category.

But the lessons of search and SEO have been well-learned by some media companies, many entrepreneurs and investors. In response, a range of so-called “content farms” has arisen to drive page views off “content” created by hundreds of mostly low-paid bloggers (and some former journalists).

Demand Media, Associated Content (now part of Yahoo), Examiner.com, Suite 101 and others recruit and train freelancers to quickly generate articles on all manner of niche topics that will drive qualified page views or lead generation in some cases. Yahoo just made a $100 million bet on this version of SEO. And AOL is pursuing a somewhat more refined version of this same strategy with Seed.

The original model arguably is “The Mining Company,” which in 1999 became About.com and was later acquired by the NY Times for its SEO/link value. While About.com was originally about “content curation” (organizing and commenting on links), the content farms are about content creation.

Despite the protests of the companies themselves about the terms “content farm” and “content mill,” the truth is that the articles and columns produced by these operations are of variable quality at best. And in some cases the content they generate should be considered a form of spam.

The proliferation of dubious or low-quality content from some of these sites is, over time, a direct threat to Google in my view. Google doesn’t present any publisher “branding” in search results so users must often click back and forth to find a quality source for the information they’re seeking. Google’s algorithm is supposed to address issues of quality and authority — and it often does — but the rising tide of mediocre, SEO-driven content creation is a fundamental problem for searchers.

Enter Blekko.

Over the past several years there have been many runs at Google and general search, including Powerset (acquired by Microsoft) and the ill-fated Cuil. None of these independent challengers has succeeded to date. (The jury’s still out on Bing of course, but arguably it has already succeeded by several measures.)

For those who haven’t heard of it, Blekko is a general search engine that will launch relatively soon. It has been written about several times by TechCrunch and I wrote about it briefly a year ago after an initial demo from founder Rich Skrenta and Mike Markson. Danny intends to do a “deep dive” on Blekko shortly so I’ll let him discuss features in depth.

There are two general characteristics that differentiate Blekko: transparency and user control. It’s also social in interesting ways; registered users can “follow” one another.

Blekko’s motto is “Slash the Web.” The centerpiece of that slogan and its chief innovation is the concept of “slashtags.”

Slashtags allow search personalization and filtering through the creation of mini-indexes of authoritative or favorite sites. For example, Skrenta has created a slashtag for wine blogs that he likes: “/skrenta/wineblogs.” In short this allows him or me to search an authoritative or personal sub-index of the internet for wine-related content and recommendations.

This way I can get articles and commentary from sites I trust or that people I trust recognize as authoritative — and cut out the, pardon the expression, crap content.

While there’s a short learning curve slashtags are easy to use and their value is almost self-evident. Blekko can also be used just like Google without slashtags as well.

Slashtags address the content-spam problem I described and provide control over results not offered by Google today. My prediction is that sophisticated search users will immediately be drawn to Blekko for the personalization, SEO tools (I’ll let Danny discuss) and social features it offers. I also predict that after it formally launches we may see some slashtag-like development from Google.

In 1998 the web was ripe for Google. But Google’s profound success and the way it has shaped the internet, giving birth to content farms, may have paved the way in 2010 for Blekko.

NOTE: Please see our detailed review that’s now available, Blekko: New Search Engine Lets You “Spin” The Web.

Google Suggest & Climategate

Let’s start with supposedly the most damning evidence first. A system called Google Suggest automatically displays search topics that it believes you may be after, when you start typing any word. So consider what happens when you start typing in the word “climategate” into Google:

In the screenshot above, when you get to the fourth letter in climategate — clim — rather than suggesting the full word “climategate,” Google instead suggests “climate change” at the top, then some other words and “climate change facts” at the bottom.

This screenshot was taken on Saturday around 1pm Pacific Time. At that time, even entering the entire word “climategate” would not get Google to suggest it. Clearly Google has an agenda!

Then again, if Google Suggest is based on popular search topics, maybe Climategate isn’t that popular. Perhaps that’s why it’s not showing. Maybe. But if that were the case, then what was happening earlier this week? Here’s a screenshot from December 2:

Notice in that, after typing in “clim,” you get a suggestion for “climate gate scandal.” Three days later, that’s gone. And back at the end of November, we had various reports that typing in “cli” would bring up “climategate” itself, which currently doesn’t show.

Official: Google Suggest Terms Come & Go

So what’s up? When I asked Google about this previously, for my Of Climategate, Googlegate & When Stories Get Too Long story, I was told:

Google has not ever removed the query [climategate] or variations of the query from Google Suggest.

Google Suggest uses a variety of algorithms in order to come up with relevant suggestions while the user is typing. We do remove certain clearly pornographic or hateful or malicious slur terms from Suggest.

That wasn’t particularly satisfactory, as it didn’t address why something my show one day and disappear the next. My best explanation was:

My assumption is that on one day, if a lot of people were searching for Climategate, then that might appear. Then if queries dropped off, the suggestion might go away. Then return again if more started searching again. I’m checking to see if I can get more clarification.

For this story, I did get a fresh statement:

Google suggestions are based on aggregate data including popular searches that have been entered on Google over time. In addition, Toolbar shows queries that a user has typed before, which are retained on the user’s machine. It is perfectly normal for suggestions to appear for a short while, stop appearing, and then start appearing again.

Unofficial: Google’s Flaky & Still Assumes Everyone Trusts It

OK, I’m sure that’s not going to please any cynics out there. But I still don’t believe Google is deliberately removing that term from Google Suggest. Having covered Google since the company literally began, all I can offer you is the real explanation that I personally believe goes something like this:

We’re kind of flaky about things at Google. We have these algorithms that you’d think should operate consistently, but they’re not perfect. So why’s that term coming and going? We don’t really know. We’d drill down into it, but we’re kind of busy building cool things with Lego. Plus, we don’t really think it’s that big of a deal. Only crazy people would think we’re really trying to manipulate people in this way, right?

That’s a typical Google failure. No, it’s not just crazy people. It’s people who have a general mistrust of any big organization. And when you’re dealing with a story where there’s evidence of a concerted effort to suppress information, yeah, some people are going to get paranoid about how the biggest information dissemenator on the planet — Google — is acting in relation to that. So put away the Lego and spend some time ensuring that Google Suggest isn’t operating as if you simply throw dice each morning to decide what it will say.

Conspiracy Theorists, Meet Common Sense

Meanwhile, those of you with mistrust of Google over Climategate? Let’s again use some common sense and examine just how well this supposed Google Suggest conspiracy is working.

First, what’s happening with Google’s competitors? Over at Yahoo, they have their own version of Google Suggest. When I start typing, when I get to “clim,” Yahoo shows me this:

That’s right, along with “climate change,” you now get “climate gate.”

Over at Bing, the same thing happens and faster, if you only enter “cl”:

Those are screenshots from today, but both Yahoo and Bing operated exactly as shown when I also looked on December 2. So kudos to them for consistency. Kudos, I suppose, for getting a term suggested that I have no doubt is popular. And if they both show Climategate as a suggestion but Google doesn’t, perhaps that’s a sign that Google is doing some editing?

Forget Suggestions; What About The Results?

Perhaps. But then again, who the hell cares what’s suggested as a search? Seriously, think about it. Do we think people who don’t know about Climategate are being prevented from learning about it because it doesn’t show when they start typing the letters of it? That makes absolutely know sense. If you’re typing Climategate into the search box, you already know about it. So what happens in reaction to your search? Let’s see:

Those are the results you get back for climategate on Google. There are far more “pro-Climategate” than anti, with pro meaning those who view the leaked emails as evidence of a concerted effort to push global warming as being manmade despite evidence from some that this might not be the case.

If Google’s trying to climatewash Climategate, why wouldn’t it skew the results you get when searching for it. So much for those boffins at Google. They’re so busy building Lego that while they rigged the suggested queries, they failed to clear out any anti-global warming articles from the far more important search listings.

Speaking of global warming, if you’re a believer that human activity is a cause of it, you’re more likely to be searching for global warming as a search term than Climategate, aren’t you? So if there’s a conspiracy to suppress opposition views, wouldn’t you do it for searches on global warming?

Let’s start first with what Google suggests for that:

As you can see, we get “global warming” (which could be pro or con on the issue of whether it is “real” or not), “global warming facts (again, pro or con) and “global warming hoax.”

Not too smart, Google. You’ve allowed those searching for global warming to understand that this might all be a big hoax.

But what about the actual results? What do you get on a search for global warming? Is there a climatewash there?

Those are the top results today. In the news section, you’ve currently got one article featured on the Climategate scandal saying that it does not disprove global warming, followed by one about global warming skeptics and one that seems against global warming concerns. That’s a fairly diverse mix.

Further below, we get Wikipedia, the US EPA site and GlobalWarming.org, which is an anti-global warming site:

Match Counts That Mean Nothing

My favorite part in all this is how Climategate proponents keep pointing at the number of results you get on Google, versus global warming, as somehow proof of how “popular” Climategate is. Let’s take James Delingpole of the Daily Telegraph, who wrote earlier this week:

Meanwhile at Telegraph blogs, the site that popularised the word Climategate – 25 million Google hits so far – there are those who just can’t see what the fuss is about.

I’d previously written in Of Climategate, Googlegate & When Stories Get Too Long about how those counts are largely meaningless, but now let me spell it out in excruciating detail.

In a search for climategate on Google, I currently get “about 30,700,000″ matches, even more than what Delingpole found:

In contrast, a search for global warming brings up “about 10,600,000″ matches:

So Climategate is three times more popular than global warming!

No. See, as I previously explained, searching for any word without putting a plus symbol in front of the word (or phrase) means that you are searching for that word PLUS other words that Google considers related to it.

A search for +climategate brings back “about 2,260,000″ matches:

A search for +”global warming” brings back “about 9,840,000″ matches:

In short, the count for global warming hardly changes but the count for climategate plunges from 30 million to 2 million. What’s going on?

Again, I asked Google. They know all the words that a broad search on Climategate would be matching. However, Google didn’t seem that interested in itemizing these, saying:

Google’s calculation of the total number of search results is only an estimate. We understand that a ballpark figure is valuable, and by providing an estimate rather than an exact account, we can return quality search results faster.

Blah, blah, blah. Translation:

Yeah, we’ve known for years that our counts make no sense, but we’ve got better things to do than to spend time improving the counts we show. I mean, check out this cool Lego thing we built! Besides, only crazy people every pay attention to things like match counts.

Is my translation too mean? No. Go back to 2006, and Google’s idiotic counts were the top item in my 25 Things I Hate About Google piece. I wrote:

1) Web search counts that make no sense.

“Why do search engines lie?” has Robert Scoble recently poking at this, on how the reported counts don’t always match reality. Heck, try class two contributions with “about” 59,800,000 matches. But then you find that only 879 are considered non-duplicates! Meanwhile, mars landing sites gives 1,050,000 matches while mars landing sites earth gives nearly double that amount, 1,840,000 listings. It shouldn’t. Adding that extra word should give you a subset of the original query. It should come back with less results, not more.

I know, I know. It’s a bug, or search counts are hard to do, or they do say “about.” I know, they aren’t the only ones, nor have they been the first (see Questioning Google’s Counts, Danny & Tristan Talk About Link Counts, Site Counts & Index Auditing and Who’s The Biggest Of Them All?). Long experience in knowing the counts don’t add up has perhaps left me numb to the issue. And goodness knows, I don’t want a return to page counts on the home page.

But then again, if you are going to put out a number, perhaps it should be accurate?

Climategate Or Climate At The Golden Gate

That’s OK, Google. No worries if you can’t be bothered to explain the crappy search counts that you put out, which make no sense and which are being quoted by major newspapers to prove how popular Climategate is. I’ll do it myself, to see if I can figure out why that Climategate count changes so dramatically.

Consider this search:

That’s where I searched for +climate +gate -climategate -”climate-gate”, which means, in order:

• Find all pages that say “climate” on them and also say “gate” on them then
• Remove all pages that have the word “climategate” on them
• Remove all pages that have the word “climate-gate” on them

What do I find? I find 10 million pages out there that have the words “climate” and “gate” on them but not “climategate” or “climate-gate.” Pages that would have existed before Climategate was dubbed Climategate. That’s a lot of pages. Here’s one about the climate near the Golden Gate:

Here’s one about climate and automatic gate openers:

My assumption is that when you search for “climategate” on Google without a + symbol in front of it, you’re pulling back some false matches like this.

Now I’d try to show this. I’d like to go through those 30 million results for “climategate” and show how some of these false matches are included. But if I try to drill into the results, I run into the standing problem that major search engines only let you see the first 1,000 results:

In that example, I’d gone as far as I could, to result 822, then chose the show omitted results option and then couldn’t get past 924 (over at Bing, I can’t get past .

As I said, no major search engine will show more than 1,000 results for any query, even if they have more. Not Google, not Yahoo, not Bing. But Google should have let me view to at least the full 1,000 (at Bing, I couldn’t get past 818). Still, even if I could get there, the ranking system will still rank pages with “climategate” on them above those that must have the words “climate” and “gate” somewhere on them.

Want Popularity Figures? Meet Google Trends

Let me be clear. I DO think Climategate is a popular search topic right now. I just think those who feel there’s a Google conspiracy to suppress it are grabbing on to whatever facts they can find (Google Suggest, Google match counts) even if those are the wrong tools to document popularity of a search topic.

Folks, you want Google Trends. That shows you the volume of searches that have happened over time. As topics get popular, there are more searches.

Let’s see how climate change, global warming, climategate and climate-gate all rank against each other using Google Trends:

You can interact with the chart above yourself here. It shows that for all the years that Google has search data that it shares, for all regions of the world, that global warming is the more popular term than climate change. It also shows that climategate is more popular than climate-gate.

So, in a face-off, how does global warming fare against climategate? Here’s a look for the last 30 days:

Climategate has about 40% of the search popularity of global warming, I’d say — which is a lot. There’s no doubt people are interested in this aspect of the global warming debate. Interestingly, Sweden and Australia top exhibit the most interest in it:

That brings me back to Google Suggest. It’s different for different countries. In Sweden, while Climategate isn’t suggested, “climate scam” is:

In Australia, “climate change skeptics” is suggested:

In Summary

Overall, there’s no doubt that Climategate is a popular topic, no doubt. However, those who want to demonstrate how popular would be better advised to use Google Trends, rather than the far less dependable web search results counts.

As for Google, I’ll wish again that they’d provide better results counts. I’d also hope for more consistency on how, when and why it shows suggested terms. Finally, I’m still hoping that Google will show precisely what it searched for when it looks for more than the word you’ve entered. Last year, Google grew more transparent about how it customizes results but failed to deal with broad searching as part of that. Clearly, that type of disclosure is overdue.

Back to synonym searching in a moment. The main focus of today’s post is how small changes can have a big impact on how users interact with search results. Some before and after screenshots of various search results are shown. Most compelling to me was how by making the + symbol thicker in a "Plus Box" that allows for stock quotes to be shown for a listing, usage went up. But the post also highlights how that doesn’t mean it’s an improvement. More usage might mean people are missing other important information in favor of a new gadget. Such is the thoughtful insight that goes into measuring even little changes like this.

Midway in the post, we’re told this:

The algorithm that is responsible for the titles and snippets of result pages now highlights stems and some synonyms of the original query term. For the query [hp printer drivers] we will also return results that include and highlight the word "driver".  This sort of "stemming," as it’s called, is generally a good idea, because it helps you better identify results that match your query, but not always. Experiments of this sort help us verify (or, occasionally, overturn) our assumptions regarding changes in these algorithms.

Stemming on Google isn’t new. The company has been doing it since back in 2003. It was also highlighting stemmed words back then. If you searched for "running," and it found a page with the word "run," the word "run" would be bolded in the search listing description.

So highlighting stemmed words isn’t new. What is new is that Google now goes beyond simple stemming and does synonym matching.

For example, with stemming, a search for running might match:

• run
• runs
• runners

But it wouldn’t match:

• jog
• jogs
• jogging

This is because the word run doesn’t make up part of those words (stemming means that the word you searched for makes up the "stem" of other words).

With synonym searching — or concept searching or thesaurus searching, which are other terms used — you do a search to find matches for the original word, stem variations of the word, and words that are conceptually related to it.

Google doing searches for synonyms is a big change and one I wish wasn’t buried in the middle of this other post. It deserved highlighting on its own, or in another posts where it was mentioned, such as here in July:

It is critical that we understand what our users are looking for (beyond just the few words in their query). We have made several notable advances in this area including a best-in-class spelling suggestion system, an advanced synonyms system, and a very strong concept analysis system.

I remember reading that briefly in the middle of a trip and flagging it for follow-up. Google’s doing conceptual search now? When did that happen, I thought.

Anyway, it’s official now. I pinged Google for a bit more info, and they sent:

Yes, the highlighting of synonyms has changed. Also the degree to which we understand synonyms changes and improves, and this can affect the impact on highlighting the terms. The main point is that the algorithm does change. We mostly highlight stems, not synonyms, but the distinction is lost on most people."
 

One thing I hope will come soon after this highlighting is a flag to let people know if their search as been customized to match stems or synonyms. It can make a difference to searchers, and they may not realize it happening now.

Google Now Notifies Of "Search Customization" & Gives Searchers Control from last month covers how Google does such flagging in other cases.

What’s Normal?

There was a time when everyone saw exactly the same search results at the major search engines. Over time, this has changed. In particular, geographic targeting has meant that searchers in different countries often see different results (see How Search Engines Redirect Users To Country-Specific Sites for more on that). In addition, personalized search results on Google mean that, more and more, searchers are seeing things differently than what others get. "Normal" search results are becoming an endangered species.

Still, there are times when you want Google to get back to as normal as possible. Today’s rollout is designed to inform searchers when Google is customizing their results in three key areas and allow them to regain control. Note that this only works for English queries on Google.com, at the moment. Not everyone will see it immediately, either — but it should go fully live over the next few days.

Geographic / Location Customization

For some queries, Google may try to alter your results based on your geographic location. It makes sense in many cases. Johanna Wright, director of product management at Google, gave an example of this.

"If you’re in Seattle and type in the query ‘zoo,’ we’re going to show you the Seattle Zoo rather than the San Diego Zoo," she said.

Nevertheless, Google doesn’t always get it right. If you’re looking for local information but outside the area you’re interested in, you might want to override Google’s customization. Now you can.

When customization is happening, you’ll be told through a new message at the top right of the results that says "Customized for" followed by the location Google assumes you’re located in:

Next to that is a "More details" link. If you click on that, you get a new "Search customization details" page where you can change your location:

If you’re not logged into Google, your location is determined by your IP address. Selecting the "Use a different location" option will override the guess Google makes and be remembered as long as your browser accepts cookies.

If you are logged in, Google will use the location you’ve saved as your default in Google Maps. Not sure how this is done? It took me some time to figure out myself!

By default, Google automatically saves the last 100 addresses you search for into a "Saved Locations" list, something added back in 2006. You access this by using the drop-down option next to the search box in Google Maps. Do that, select "Edit Saved Locations," then click on the green arrow next to the location you want stored as your location.

Never set an address? Then Google will guide you through the process to do so, if you try to change your location when logged in using the search customization page.

Previous Query / Recent Searches Customization

In April, we reported how Google was beginning to tailor search results based on your current search plus the search you just did. Called Previous Query, it’s where Google might see that you first searched for [hotels] then did another search for [new york] and so will effectively combine the first search to the second to make [new york hotels]. You’re searching for all three words even though you only put in two — new york — on your most current query.

This is another feature that can be useful in some circumstances but where the searcher might want to override it — or just understand when it is happening. Now if previous query kicks in, a message saying "Customized based on recent search activity" will appear at the top right of the results. Clicking on "More Details" link from that message will take you to the search customizations page, where you can then discover what previous query was used.

Via that page, you can also reissue the query without the previous query being involved. Nice, but it would have been better if way back on the search results page itself you could simply click on a link to redo the query without customization. I can understand one reason this isn’t happening. Most searchers probably would want to know what the customization was before overriding it — which means showing them the actual previous query. And there could be privacy issues in that.

Indeed, Google is addressing some of these issues already For example, if you do a search, leave your computer and someone else sits down and does a search, they might get their new query customized based on your previous one. And if they click on the link, they can explicitly see what you just searched for.

To minimize this concern, Google only makes the last query performed visible for 30 minutes. And if you close your browser, it can’t be seen at all. While I’m sure some will still have concerns about this, keep in mind that most browsers themselves will automatically store an entire long history of past searches that were done — far more revealing and accessible to anyone who wants to snoop around.

Note that the customization option only impacts the unpaid results you see. Google also shows paid listings that also use previous queries. Indeed, Google Looking At Multiple Previous Queries To Tailor Search Ads covers how several queries "back" might be used to customize the ads that are shown. The new search customization option doesn’t control what shows here.

Web History / Personalized Results Customization

Since last year (see Google Search History Expands, Becomes Web History for more), Google has heavily pushed the customization of search results based on the places someone visits on the web, past searches they’ve done, what they’ve bookmarked with Google Bookmarks and what they save to the iGoogle personalized home page.

When results have been personalized, Google’s shown a "Personalized Results" message at the top right of the page. Now that’s being enhanced to say search results have been "Customized based on recent search activity" along with a "More Details" link to control this. Click from that to the search customization page, and you are then invited to either remove some items in your web history ("manage") or remove web history entirely.

I wish next to the More Details option that Google would have also just placed a "Pause" button, so that personalized results could be paused and "normal" results viewed easily, without requiring a log-out. And if someone stayed on Pause for a long-time, Google could add a further message like "See Personalized Results" when appropriate.

Being Transparent

Why do all this? Google’s pitch is that it’s part of trying to be transparent with searchers, both to help reassure them about their privacy and let them know some of what’s going on under the hood to generate their results. Indeed, Wright said some of the push for showing these options came from power searchers at Google itself.

"We had a couple of power users at Google who were complaining that they didn’t know what was happening," she said.

I can understand that and appreciate having more of this being revealed. There’s a long history of search engines changing how they perform a query from what a searcher entered, so telling searchers when this is happening and how they can override it if they want is great.

What’s next in terms of control? Wright couldn’t reveal, though I think a likely candidate might be stemming, when Google automatically searches for variations of a word in addition to the word itself.

Intriguing would be if Google really exposed even more factors. Are results being skewed in some cases toward non-commercial sites? If so, could the searcher choose more commercial listings, if they want? Could a searcher explicitly nominate sites to have higher PageRank values than Google itself gives them? If so, that might let them see a different view of the web they might prefer. Are fresher results getting a short term boost in a query? Perhaps a searcher could be told that, so they can override that decision if they want.

Below, the examples are drawn from someone in the UK trying to reach .com versions of each of the major search engines. Those in other countries often will find similar interception in place.

Yahoo

If you are based in the UK, Yahoo will now redirect you to Yahoo UK & Ireland. The UK site shows the Yahoo UK & Ireland logo but also has a top banner explaining the new behavior. Search Engine Land editor-in-chief Danny Sullivan — who is in the UK — sent me a screen capture and explained the behavior of the feature. The screen in the UK looks like this:

Yahoo, right off the bat, tells you your Yahoo experience will be customized based on your locality. If you do not want a locally customized experience, click on "Go to yahoo.com" link, and you will be taken to the US version. Once you do this, the choice is remembered and stored via a cookie. The next time you try to go to Yahoo.com, you can go there directly without being redirected to Yahoo UK.

It is important to note that Yahoo’s results, even if you use Yahoo.com, will be skewed toward what it believes those in the UK would like to see. This will happen EVEN IF you DO NOT use the "UK only" box. The other major search engines also do such skewing. For Yahoo, we don’t know any way to override this behavior.

Google

Like Yahoo, try to go to Google.com in the UK and you’ll be redirected to Google UK. Also like Yahoo, you can override this. You use the "Go to Google.com" link at the bottom of the page. Do this once, and your choice is remembered for the future. Redirection will no longer happen (unless you clear your cookies for some reason).

Also as with Yahoo, results and ads on Google.com will still be skewed to favor what a UK user may want. There is a way to override this for ads and, to some degree, for the general search results. After you do a search, add this to the end of your query:

&gl=us

For example, if you searched for football, you’d see this in your browser’s address bar:

http://www.google.com/search?hl=en&q=football&btnG=Google+Search

At the end of that, insert the gl code, like this:

http://www.google.com/search?hl=en&q=football&btnG=Google+Search&gl=us

Now the results will be more US orientated. Want to see how they look for other countries? Instead of US, use the two digit code for any country you find listed over here. You can also use the AdWords Preview Tool to see locally based Google search ads.

Live Search

Live Search does NOT redirect you to a new URL. Instead, it gives you a customized look and localized search results for your country. The screen capture above shows the option for "Only from United Kingdom" results, since it was snapped from within the UK.

In the advanced options, there’s an option allowing you to set your location for anywhere in the world. However, Danny says that hasn’t worked properly for at least a year.

Fortunately, those wanting to see US results (or those from any country) can use a brand new Live Worldwide page he was just told about from Microsoft this week. The page hasn’t even been posted to the Live.com site, so you’ve got an exclusive to play with it.

Ask.com

Ask will redirect you to Ask UK. There is a link at the very bottom of the page that says "Ask.com US." Click on that to get to Ask.com. Unlike Yahoo and Google, this isn’t stored. You’ll have to do it each time you want to go to Ask.com from your browser address bar.

With the US about to celebrate the Thanksgiving holiday tomorrow, how are the major search engines gearing up to support an onslaught of Thanksgiving queries? Let’s take a look at how you can use them, ranging from flight tracking tools to special "shortcut" or "smart answer" boxes leading to related information.

Many people travel by air on Thanksgiving. Want to know if friends or family are going to make it on time? Several of the major search engines prominently list flight tracking information, as well as airport conditions and related information. Do the right search, and you’ll get special links at the top of the pages, like this, showing flight tracking at the top of Google:

Here are traffic conditions on Yahoo:

And here are airport conditions at Ask:

How to make these special links come up?

• Google: Enter the name of an airport or a flight number to get relevant information or links at the top of the search results. Weather info is also offered. More is explained at Google help here. Some examples: aa123, ohare airport, weather los angeles.
   
• Yahoo: Get information about airport conditions, flight tracking, as well as local weather and traffic reports, all at the top of your search results. See more advice covered here at Yahoo. Some examples: aa 123, weather los angeles. traffic los angeles. NOTE: A search like ohare airport is supposed to bring up special info, but I found it didn’t, so the airport conditions feature might not be working as advertised.
   
• Ask: Get weather info, as explained here (and weather los angeles brings up an example). Airport conditions aren’t explained on that page or in the help files that I can find, but they do work for some, such as jfk airport, lax airport, but not ohare airport.

Want more flight tracking resources? See ResourceShelf’s recent post on dedicated flight tracking services, which are free and easy to use. Nothing is cooler than tracking the flight of friends, family, and loved ones as they progress to you.

Back on earth, remember that most of the major search engines have real-time traffic info available through their mapping services. Our Google Maps Tailgates Yahoo, Microsoft With Real-Time Traffic Info explains more about this, with illustrated examples. Mobile editions of the mapping tools from the major search engines also have real-time travel, if you have the right type of phone. Get them here: Google Mobile Maps, Yahoo Go, & Microsoft Live Search Mobile.

That chart above? That shows you how, according to Google Trends, searches for "thanksgiving" ramp up before the actual day, then skyrocket when it happens. How have the search engines prepared for the rise in queries?

Over at Yahoo, there’s a Yahoo Shortcut telling you the exact day of Thanksgiving in the US and leading to an encyclopedia entry hosted at Yahoo about it. There’s also a link to Thanksgiving menus and recipes at Yahoo Food:

Yahoo has also blogged about their turkey recipes shortcut that appears.

Ask also provides the date and a recipe links, but the Ask Smart Answer it runs further includes a link to history, Thanksgiving crafts, games, and more. An screenshot of this is at the top of this article.

Some of these links really just run a new search, such as the history link bringing up a search for history of the First Thanksgiving. That’s a crafty way to get people to generate more searches on Ask, but I would have preferred if they send me directly to some selected information, rather than a new search page with yet another Smart Answer box at the top.

Ask also has a special blog post up illustrating how to search for Thanksgiving information from video sources to news reports.

As for Google and Microsoft, neither have anything special going. But over at Quintura, those that light the idea of clicking on through "clusters" or topics to get search results will find a new "Thanksgiving" tab at the top of the page. Select that, then you can click on Thanksgiving-related topics to generate results.

Now to shopping. As many know, there’s heavy post-Thanksgiving shopping happening on the day after, Black Friday, and the Monday after, Cyber Monday. Related to that, Hitwise tells us:

Searches on the term "black friday" and "black friday ads" are up 174 and 143 percent respectively, compared to last week. Combined, the percent of U.S. searches on the two terms has increased nine percent compared to 2006.

and:

The iPod and the Nintendo Wii were the most popular search terms sending visits to Shopping and Classifieds websites for the week ending Nov. 17, 2007. The terms "ipod", "nintendo wii" and "wii" were the top three product searches. Video game consoles remain a popular product search so far this holiday season as the Nintendo Wii, Xbox 360, and Playstation 3 were all among the top 10 searches this past week.

Finally, tomorrow, expect to see some custom Thanksgiving logos hit the search engines.

You can translate from English to German, Dutch, French, Spanish, Portuguese, Italian, Korean, Chinese, Japanese, and Russian. It supports up to 500 characters and uses Systran, but can override the Systran source with their own in-house translations.

Google Operating System goes deeper into this translation service.

Aside from Microsoft, there’s the classic Babel Fish translation service, which Yahoo acquired. Also, Google offers translate services, which were recently upgraded, as do many other companies.

ResourceShelf noticed last week that Microsoft acquired some domain names related to this live translator service.

Mossberg gave Ask.com great reviews, comparing Ask.com’s three-pane approach to Google’s Universal Search approach. Mossberg said Ask.com “took greater strides” than Google with their user interface. The new Ask.com user interface makes for a “better advance in unifying” the different levels of vertical search options available to the searcher, said Mossberg. He explained that this makes “search results easy to navigate and use” than Ask.com’s competitors.

Mossberg’s last word on Google’s user interface compared to Ask.com’s user interface,

Google deserves credit for universal search, which I’m sure will get better. But Ask’s new design is much more compelling and well worth a try.