The core data comes from CommonCrawl, a non-profit group designed to crawl the web and provide data for anyone to use. Gil Elbaz is both a founder of CommonCrawl and of Factual, a start-up that creates tables of structured information from data found on the open web (see Factual: Parting The Curtains Of The Invisible Web).

Factual found stats such as I cited above after examining 4 million web sites. In particular:

• 28% of sites have Google Analytics on them
• 12% of sites have AdSense
• 5% of sites have EITHER a Twitter or Facebook link but…
• 2% of sites have BOTH a Twitter or Facebook link

There’s also a chart that shows other interesting stats but without precise percentages. I’ll estimate as best I can:

• About 20% of sites have Flash
• About 19% of sites have an RSS feed
• About 6% of sites have a sitemaps file
• About 1% of sites have a Google Webmaster Central verification code
• About 1% of sites have Quantcast tracking code
• About 0.5% of sites have a Creative Commons attribution

One thing unclear is how the stats break down on a page versus web site basis. A web site might have multiple pages. So when a “web site” is said to have AdSense on it, does that mean each page within the site has AdSense code? Or only some of them? It appears a decision was made on a site-by-side basis, with “site” being defined as all the pages within a set domain or subdomain.

Those interested can play with the data themselves. It’s summarized in this very large table at Factual.

CommonCrawl also gets a bit of publicity from this at an interesting time. Earlier this week, Google released a long internal memo talking about how important it was to the company to be open — except in the areas of search and ads:

In many cases, most notably our search and ads products, opening up the code would not contribute to these goals and would actually hurt users. The search and advertising markets are already highly competitive with very low switching costs, so users and advertisers already have plenty of choice and are not locked in.

I’ll likely do my own follow-up post to that memo in the near future. In the meantime, a post I wrote back in 2007 — Google: As Open As It Wants To Be (i.e., When It’s Convenient) — looks at how Google’s claims of being open tend to ring false when open isn’t something it seems to pursue in areas where it is ahead. In part from my post:

That large index gives Google a huge advantage over rivals. It knows more about what’s on the web than anyone else. So why not share? Why not start an Open Index Alliance where there’s a coordinated effort to crawl and index all the documents in the world, allowing anyone to tap into the raw data?

That’s the idea behind CommonCrawl. Maybe as part of being open, Google could get behind the project?

See also Chris Dixon’s post from this week, Google should open source what actually matters: their search ranking algorithm, for related thoughts about Google, search and openness, along with comments from me and others, including the head of Google’s spam fighting team Matt Cutts.

As for ads, see Schmidt: Someday, AdSense Publishers May Know Google’s Cut Of Ad Revenues, from me earlier this year, which looks at how most AdSense publishers have no idea how much money Google keeps back for itself. It’s hard to find an arugment that support not being open about this, in the face of Google’s declared love of open.

There was an iPhone-friendly mobile website that has been taken down to avoid the “why buy the cow when the milk is free” problem. We’ll see if the $19.99 price tag is still too high.

But if you’re inclined to jump at this lower price point . . . act now, supplies are limited. No rainchecks.

If you have opt in to the Social Search feature at google.com/experimental you will notice an error on your Google searches that reads, “The experiment you’re trying to access is no longer available. Go to experiments overview.”

Google has confirmed this is a temporary issue and that a fix will be in place sometime today or tomorrow. Here is Google’s official statement on this matter:

The Google Social Search experiment is temporarily down. We are working on it and expect to restore access sometime Monday or Tuesday.

Mayer replied: “It would be a machine that could answer that question, really. It would be one that could understand speech, questions, phrases, what entities you’re talking about, concepts. It would be able to search all of the world’s information, [find] different ideas and concepts, and bring them back to you in a presentation that was really informative and coherent.”

What Mayer may have unknowingly described is Siri, a “virtual personal assistant” that uses artificial intelligence to determine user intent and then match data or applications that can fulfill that intent. The company will launch its iPhone application soon and already has a deal with a “tier one” US mobile carrier. The NY Times offers background on Siri and some of the technology behind the system:

SRI International’s software venture, called Siri, is more ambitious, in that it allows users to speak or write natural-language requests into the device (“Find me a place to eat dinner tonight with Karen, reserve a table and put it on our calendars.”), which will complete the task independently and inform you when it is done.

In terms of long-term predictions, Siri is actually an easy bet. Dag Kittlaus, the company’s chief executive, said one of the four major carriers would introduce the service early next year, and he said it would also be available as an iPhone app. But over the next two years the technology should be able to complete a wider range of tasks.

I’ve seen Siri in action and found it impressive. The system is not perfect but it brings users closer to transactions and fulfillment of their objectives — at least in a range of use cases — than can Google on mobile devices today. It uses a voice interface to receive queries. You can use the keyboard if necessary but that’s entirely secondary to the experience. 

The way one interacts with it is “conversational” and “transactional” rather than providing a verbal version of a conventional search query. 

I moderated a panel at the recent Open Mobile Summit in San Francisco called “new directions in navigation and search.” The panel, among others, featured Siri CEO Dag Kittlaus. What became clear during the panel is that we’re going to see lots of innovation and change in mobile search and that the present version of the experience could well be regarded as Jurassic in only a few years as the unique attributes of the device (e.g., the camera) become input mechanisms and search tools. Augmented reality is also a part of this, although in its present form it’s fairly undeveloped and limited.

And, as another example of how far things could develop away from the current “query box and blue links” search paradigm,  look at the video demo below of “SixthSense” a “wearable gestural interface”:

Click here to view the embedded video.

It’s time again for another roundup of the latest and greatest search tools and search engines, and in this article, I’ll share five such sites that will answer the above questions (and more). This is the fourth in my occasional series profiling under-the-radar search tools. Links to the previous three are at the end of this article.

SearchMuffin

Look, I don’t name ‘em, I just use ‘em and write about ‘em if they’re cool. And this one is. SearchMuffin has a simple premise: Type in a keyword and choose a city from the dropdown menu, and it’ll show you the Google search results that match. Think of it as a sort of geo-targeted competitive research/PPC research tool. It’s about the easiest way I know of to see the PPC ads that appear in other cities. Here’s a screenshot of a search for “anaheim real estate” in Anaheim, even though I’m up in Washington state.

And best of all, it’s not just limited to major U.S. cities; at the moment, there are 262 choices in the dropdown menu, including such non-metropolises as Roseville, California, and Arvada, Colorado. (No disrespect intended to Rosevillites and Arvadians.)

Glearch

Let’s expand our horizons beyond 262 U.S. cities. What if you needed to quickly see some search results from other countries and/or other languages? Glearch (again, I don’t name ‘em) is an international meta search engine that lets you search by country, by language, and/or by search engine. You can take those three options and customize each to build just the query you want.

In the example above, I’m doing a search for the rock band U2, and I’m searching French-language results on the French versions of Google, Yahoo, and Bing. I’m also including local newspapers, too. The search results page has a tab for web results and a tab for news results. The web results page, like any good meta search site, tells you where each listing was found, i.e., “Ranked 1 by Google search” or “Ranked 4 by Yahoo search.”

Roooby

We’ve written a fair amount about real time search in the past few months, but we haven’t focused too much on the visual element — people posting photos and videos of what they’re doing now. Roooby is one of several real time search engines that capture media, but one of the few that surface both photos and videos. (Although, to be frank, Roooby could do a better job of finding videos by scanning sites such as Qik.com, TwitVid.io, and others that host live video.)

Roooby isn’t the only player in this space. TwitCaps, TwitPicGrid, Pingwire, and Twicsy offer similar real time image search engines.

Spezify

Speaking of media and images, here’s the most visual search tool I’ve ever seen: Spezify. The best way I can describe it is a sort of visual meta search engine. It pulls in results from Yahoo, Bing, Twitter, Flickr, YouTube, and even eBay and Amazon to create a fairly stunning search results page.

This is serious eye candy. There’s a settings page where you can choose the sources and types of content (images, text, video) you want included. But to be frank, the focus on visuals means the search results have no context whatsoever. You can move vertically and horizontally through the results, but you have no idea why you’re seeing what you’re seeing. It’s innovative to be sure, but for this searcher, it’s too lacking in functionality.

Joobili

Finally, here’s one for our readers in Europe, or for our readers traveling to Europe. It’s called Joobili, and it’s a travel/event search engine with a twist: Rather than telling the search engine what you want to do or where you want to go, you tell it when. There’s a cool date-based slider on the home page to get you started, and once you’re in the results, Joobili lets you see results based on categories (Arts, Sport, Nature, etc.), by country, or by keyword.

If you create an account, Joobili will let you save events to a wish list or a “went” list. You can also rank events to help other users make decisions on what to do and where to go. It’s a clever approach, but as I hinted above, it only covers Europe.

See also:

• Another 5 Search Tools You May Not Know … But Should
• 5 (More) Search Tools You May Not Know … But Should
• 7 Search Tools You May Not Know … But Should

The webcast was apparently difficult for some to view, but you can get impressions of what happened from Between The Lines and Bits.

The site also has a new blog and Twitter account, and we’ll be looking at it more closely in the near future ourselves.

There’s more discussion on Techmeme.

Postscript: See Impressive: The Wolfram Alpha “Fact Engine”, our review of the service.

Out now from Microsoft Research is U Rank, an experiment that allows people to move results around, as well as share them with friends and add comments to listings.

Want to play? Sigh. You have to register using a Windows Live ID. Why not just make it open for anyone to use on a cookie basis? The cookie could allow short term changes to be remembered, while logging in could be done for those who wanted to protect their edits in the long term.

Anyway, once in, you can do a search and wait, and wait, and wait to get back results. Perhaps the response time will improve. When they appear, you can then hover to the right to get options like this:

These allow you to:

• Move an item up or down (click on “move me,” then you drag to where you want)
• Copy an item to another search (this is weird, and I couldn’t get it to work. I assume it will move a listing to appear in the results of another search that you indicate)
• Add a note to any item (this also wouldn’t work for me, perhaps because I’m using Firefox).
• Delete a result

Edits that you do then are recorded and appear the next time you do a search. IE, move a result higher, and you’ll see it higher next time you search, along with a little flag indicating you did an edit.

By default, sharing of searches is on, as you’re told next to the search box:

This means any search you do is automatically shared with those you friend through the service. That’s a terrible idea, having it on by default. I think people will fail to remember to block sharing for some fairly personal searches they do. Far better to make sharing something you do explicitly, to be on the safe side. Of course, the downside to this is that people might not remember to do so, which takes away from the social experience Microsoft is aiming for with this test.

Is search really social? My Search 4.0: Putting Humans Back In Search from earlier this year argued that in many cases, I feel it is not. It also touches on the privacy issues raised with sharing:

Finally, privacy is an overlooked issue when it comes to social search. People often search for intensely private, personal things using search engines. Search engines are almost like confessionals, where people seek solutions to problems they might not tell real people that are close to them. With social search, do they have to remember to turn off a sharing feature that might be activated by default? And if it’s not on by default, will it get any take-up at all?

Of course, you start the service with no friends at all, so sharing isn’t that much a worry at the start. At the moment, once you join, others that seem to match you are somehow suggested (I had suggestions from people within Microsoft Research). You have to actually accept them, however. You can also invite two other friends, at the moment.

Friends are able to move results that influence what you see. In other words, if they do an edit, you’ll see that edit in your own results. Plus, you can make use of a tag cloud of searches to see what others are searching on.

Concerns on sharing aside, the project looks interesting. It also looks a lot like the Google Like/Don’t Like experiment that has been off and on since last year, as well as editing tools that Wikia Search rolled out and ones that I think Mahalo also has. Hakia also just recently rolled out a way for group edits in the form of “trusted results” using librarians and informational professionals to contribute.

In short, the wave of letting users edit their own results is back? Back? Yes, because if I had time, I’d add more about how we used to have many of these same tools with the major search engines years ago. But no matter — the ability to edit and move results is welcome, and I hope we’ll see more of it.

And will we see this on Microsoft Live Search? Says Microsoft in their email about it:

Live Search has absolutely no intention of implementing this…

Of course, I’m sure Live Search almost certainly WILL implement anything that they find useful on this experimental site. After all, the email goes on:

The goal of the research project is to learn more about “how” people use search technologies, like whether they take advantage of the ability to edit search results and how they share the results over time with friends and family.

So check it out. Just remember to turn that sharing off or be sure you really, really like the folks you friend on the system, if you use it much.

We have reports from TechCrunch, Google Operating System, and Justin Hileman, with reports of users seeing this feature in the main search results. Justin does an excellent job taking us through each feature with screen shots.

I personally have never seen an implementation of this on any of my searches. But I know Google has been testing this over the course of the year.

The new search engines I’m talking about are the semantic search engines, meaning they are search engines that can be queried using natural language (not keywords like when using Google). Behind the scenes, these search engines try to understand the meaning behind the text web pages and so when you query them, they map what your query means and find answers based on the meaning they’ve extracted. It’s all very neat, and there are many examples: Powerset (which Microsoft recently acquired), Hakia, [true knowledge], Cognition, and a few others.

Since these are still early days, I won’t be telling you how to do SEO for this breed of search engines—we can’t just yet for various reasons. Instead, I want to get you to start thinking about these search engines and what they mean for SEO in the future. I will illustrate my points with two simple examples.

While we’re at it, two quick notes: at the moment, these search engines are definitely still in beta and are of the quality of mobile phones about 10 years ago: they work most of the time, but they break regularly enough to remind you the technologies, and the companies behind them, are still in development. A related note is that their coverage is not great; actually, Powerset and Cognition are developing their algorithms using only Wikipedia as their corpus.

On to our two examples:

Who built the empire state building?

In January I visited the Empire State Building in New York City for the first time. I wanted to learn more about the building, so I tried this natural language query at four engines: “Who built the empire state building?” Have a look at these result pages from each engine:

• Google

• Powerset
• Hakia
• Cognition

There is a lot to analyze, and it’s interesting to compare how different these results are, both from Google and from one another.

Hakia is confused but takes an excellent stab at answering the question by referring us to the Empire State Building’s official website. This is the second best feature of Hakia in that it tries to answer you directly (I’ll illustrate Hakia’s best feature in the next example). This is the best innovation in search since PageRank, and it all started with Google’s calculator feature. So our first SEO point is how do we get to be the answer for this kind of functionality? As an aside, True Knowledge (mentioned above) is actually much better at this, in that it directly answers queries all the time, and (amazingly) can deduce facts it doesn’t know from other facts in its database (see their demo video or ping me if you would like an invite to the beta).

Powerset answers the question immediately using its Factz feature. The person responsible is apparently John J. Raskob. Clicking through to the Powerset result (which is a Powerset-hosted Wikipedia copy), we can see the highlighting of the sentence from which Powerset derived the answer. Very cool! Hakia and Cognition also do this highlighting of results.

The Cognition highlights their first hit is probably the best as it’s quite thorough. It also highlights sentences that take a wide interpretation of the verb "build," highlighting sentences talking about the designers and the construction company. Very, very neat, and raises an interesting question: if these search engines have a cached copy of our content so they can highlight the answers, will they ever send traffic to our sites? Think about it from the user’s point of view: why leave a search engine’s super-functional “result” page when you already have the answer?

Look at Google’s results: they are nowhere near as useful as Powerset’s or even Hakia’s best-shot. Also, the number one hit does contain the right answer, but the fact that it’s number one seems to be because it exactly matches the query, not because Google understood the query or the page. Thus, our third SEO point: you can optimize a page to respond well to keyword-based queries and natural language queries.

Next question:

Who makes Diet Coke?

This question is interesting because it has a specific answer and shows a range of responses from the search engines:

• Google

• Hakia
• Cognition
• Powerset

Google and Powerset win this round hands down, Google because it can understand synonyms and parse the structured data in Wikipedia, and Powerset because it understands the meaning properly. The highlights in Powerset’s result show clearly Powerset’s algo in action. It works really well.

What else can you do to start thinking about doing SEO for semantic search engines? Consider these points:

Did you notice how the highlights were for sentences or fragments? To me, that tells me that these search engines are analyzing text in those terms. Another clue comes from Powerset measuring its analysis performance in seconds per sentence, at one second per sentence about a year ago. So when you write content optimized for them, think about how each sentence can answer a specific query related to the page. If content was previously king, it’s now emperor!

Think about reputation management. Is Coke good for you? Who is Bill Gates? (Answer at Hakia, and that’s their best feature; they call it the Gallery). Facts about brands and people are straightforward to discern. Imagine a semantic search engine highlighting a sentence that says “Coke has been shown to be bad for kids.” It doesn’t need to be accurate, but it’s an answer nonetheless.

Which raises this question: how will these search engines determine authority? Once they leave the confines of Wikipedia, which is reasonably accurate, and begin to index the full web, will they return sites in their results simply because a page happened to have the right sentence structure to answer a query? Sadly, from my experience, Hakia seems to be doing this already. Not being able to accurately identify authority could be deadly for the success of these semantic search engines.

What about links? We’ve been link building for years and so far not one of these search engines has talked about links! What on Earth will we be doing in the future? It could still be links as a way to measure authority, but it could be something else. Watch this space!

One way or another, semantic search engines will be part of the future of search engines in terms of natural language queries and indexing. This is new to our industry and we have to sit up and pay attention. Failure to do so may mean that you will miss the next big thing. On the other hand, all these search engines could go bust and we remain stuck with our keywords for a while longer. You choose, but to me, the answer is clear.

Pierre Far recently launched Social Alerter and maintains a set of SEO tools. He has a PhD from the University of Cambridge, UK, and works as an innovation consultant.

I looked at Searchme, Viewzi, and PicLens, all of which have been reviewed recently in various places. For each, I purposely went into them without reading any of the help information. As a result, I might be missing out on some cool features or capabilities they have. But then again, so too are the typical people these services hope to attract. No one is reading how to search at Google. Even fewer than no one will read how to use these places.

Searchme

At Searchme, when you enter a search query, immediately below the search box, categories appear. The exact categories change depending on what you search for. For example, for dvd player, these options show up:

• luggage & bags
• software
• movies
• business news
• photography

Selecting one of the category icons, rather than doing a general "All" search, narrows your results to pages deemed to be relevant to that category. This is both cool and weird. Luggage as a "dvd player" category? It turns out that the results coming back by doing this, showing DVD player cases, were indeed interesting. But DVD players narrowed by the Martial Arts category? Not so compelling!

More importantly, most people probably won’t use these options. There’s a long history of users ignoring links like these around the search box, which tends to be like a black hole that sucks users in.

How about the main attraction, the "visual" search results:

Sure, they look cool. But as is often the case, "cool" doesn’t mean useful. The screenshots of each page are fairly hard to read — in some cases, impossible to do so. That means you’re having to judge whether a site matches what you want almost entirely based on what it looks like. Consider how "useful" this is for Sears:

OK, you can get a description for each page if you know to hover your mouse over it. One will appear at the bottom of the image. But that leads to another flaw. Which is easier — to quickly scan 10 textual descriptions or to painfully click-click-click your way through a screenshot at a time?

The "stacks" feature is pretty neat. You can drag any screenshot to the top left of the page, where you can make a collection of search results. Of course, Microsoft offers this for image search. And they even have it on their own "visual" site, Tafiti. Plus, over the years, we’ve had other search engines offer a drag-and-collect feature for web search. It still hasn’t taken off. I do think it’s a great idea, and it would be nice to see it come to places like Google. But Searchme Stacks is hardly a killer feature.

At Searchme, you can also narrow searches to just video or images. Here, the visual display is more compelling. But still, I think it’s easier for the searcher to review a lot of pages at once using the "old school" listing model.

Searchme is also slow. It takes noticeably longer to get back results than on a major search engine, and there’s no great payoff for that wait. The Flash-based service also crashed my Firefox 3 setup once. OK, Firefox 3 is new, and I’m not using the final release candidate. Nevertheless, none of the major search engines, including Google, have crashed it once.

Viewzi

Like Searchme, Viewzi is slower than a regular search engine. Even worse, after doing my DVD player search and waiting for a response, when it came, I then had to decide what "view" I wanted to see:

The screenshot above only shows a slice of the many "views" that are out there. Working through them was somewhat overwhelming for me, and I write about search all the time. A typical person hitting this page is being asked to decide if they want:

• Video: Blinkx, Veoh, YouTube
   

• Site Information: Alexa, Delicious, Google, Summize
   
• 3D Photo Cloud: Flickr
   
• 4 Source: Ask, Google, Yahoo, & Microsoft MSN Live Search
   
• and more…

Assuming you figure out the choice to make, it gets worse. Consider if I just want to see the results of the four major search engines at once:

You don’t know much of anything about what’s showing up. After studying the page (something a typical user isn’t going to do), I realize that the first result in the top left corner is showing up because it is listed on all four major search engines (thus the "stacked" look to icon, a page for each of the search engines it is on). But the purpose of also showing the lower row? Plus, I kept having a problem where if you click, sometimes I’d get a larger preview of a page but other times I’d be jumped out to it. This is a user interface nightmare.

Maybe if I were to dig further into some of the "views," I’d find an example of a compelling reason for this display. Something like Album View (check it out for The Weepies) is intriguing. Viewzi might have more success if it focused on a few particular views where visual display is really useful. But right now, it’s just a search engine relying on visual gimmick that people don’t need.

PicLens

Quick history lesson. How many search engines have been successful by requiring people to download software before they can search? Zero. That’s bad news for PicLens, since it assumes it can swim against the tide.

Let’s assume you do download PicLens. The next mystery is figuring out how it works. I fired it up on Firefox, then after Firefox restarted, I was clueless what to do next. Remember — I’m being a typical user who doesn’t read the help files.

Eventually I noticed a new icon next to the Firefox search box in the top right-hand corner. Clicking on that caused an entire new window to appear, sort of freaking me out.

I ended up having to go to the demo page to better understand how the tool worked — which to me is a failure on the search usability side.

PicLens probably produces the "prettiest" visual results of any of the tools I’ve covered. But again, I couldn’t see how doing a "visual" search for "DVD player" on Amazon using PicLens increased the search experience. Rather, it just slowed things down.

Save Me From The Future

Clearly, I wasn’t impressed with the visual offerings. It’s not that I don’t like cool things. It’s just that there should be a reason to display things visually. It shouldn’t just be an excuse to look different. Moreover, text IS visual — and the textual display metaphor continues to be used largely because it does work. But for particular types of searches, a more graphical display can make sense. Tying local search to maps is a classic example of this.

I am looking forward to a more visual search future — but don’t make me fly through results unless it helps me to do so!