Recently IDG News Service asked Google’s Marissa Mayer about the “perfect search engine.” Here was the question posed: “What is the perfect search engine? If you had a magic wand and could create it, what would it look like? What would it do?”

Mayer replied: “It would be a machine that could answer that question, really. It would be one that could understand speech, questions, phrases, what entities you’re talking about, concepts. It would be able to search all of the world’s information, [find] different ideas and concepts, and bring them back to you in a presentation that was really informative and coherent.”

What Mayer may have unknowingly described is Siri, a “virtual personal assistant” that uses artificial intelligence to determine user intent and then match data or applications that can fulfill that intent. The company will launch its iPhone application soon and already has a deal with a “tier one” US mobile carrier. The NY Times offers background on Siri and some of the technology behind the system:

SRI International’s software venture, called Siri, is more ambitious, in that it allows users to speak or write natural-language requests into the device (“Find me a place to eat dinner tonight with Karen, reserve a table and put it on our calendars.”), which will complete the task independently and inform you when it is done.

In terms of long-term predictions, Siri is actually an easy bet. Dag Kittlaus, the company’s chief executive, said one of the four major carriers would introduce the service early next year, and he said it would also be available as an iPhone app. But over the next two years the technology should be able to complete a wider range of tasks.

I’ve seen Siri in action and found it impressive. The system is not perfect but it brings users closer to transactions and fulfillment of their objectives — at least in a range of use cases — than can Google on mobile devices today. It uses a voice interface to receive queries. You can use the keyboard if necessary but that’s entirely secondary to the experience. 

The way one interacts with it is “conversational” and “transactional” rather than providing a verbal version of a conventional search query. 

I moderated a panel at the recent Open Mobile Summit in San Francisco called “new directions in navigation and search.” The panel, among others, featured Siri CEO Dag Kittlaus. What became clear during the panel is that we’re going to see lots of innovation and change in mobile search and that the present version of the experience could well be regarded as Jurassic in only a few years as the unique attributes of the device (e.g., the camera) become input mechanisms and search tools. Augmented reality is also a part of this, although in its present form it’s fairly undeveloped and limited.

And, as another example of how far things could develop away from the current “query box and blue links” search paradigm,  look at the video demo below of “SixthSense” a “wearable gestural interface”:

You like it! You really like it! It being the Google Search Options feature that Google added to its search pages back in May, the company says. So as a reward, Google is adding options to filter out (or highlight) pages you’ve visited before, pages that Google’s found in the past hour and a toggle to show more or fewer “commercial” listings in its results.

My companion piece, Up Close With Google Search Options, does exactly as the headline says. It will take you through ALL the Search Options that Google’s offers in great depth, including the new ones. But for those who just want to hear about the latest features, here you go.

Visited / Not Yet Visited Filtering

Follow the arrow in the screenshot below:

See that new section? It shows up if you’re logged into Google and making use of its Web History feature (see Google Search History Expands, Becomes Web History for more about this).

When you’re logged in, you can select the “Visited Pages” option for any search to see only the pages that you’ve been to already via past searches on Google and related to words you’re searching for. You can also do the opposite — filter out any pages you’ve already visited for a particular search, so you can focus on new areas to explore.

More / Fewer Shopping Sites

At the bottom of the Search Options panel, you’ll see the results section:

This area features two new options: “fewer shopping sites” and “more shopping sites.” These do exactly what’s described. Click on fewer, and if you search results had too many shopping results, they go away. Think of it as the Amazonator. Like shopping sites? Well have more of them! Just use the “more shopping sites” option.

Pages From The Past Hour

As the screenshot above shows, a new “past hour” option has joined the existing date filtering choices of past day, past week and past year — as well as the customer date range choice.

Not enough for you? Our Hidden Google Feature: Find What’s New In The Last Minute Or Second covers how to drill down even more. But also see our What Is Real Time Search? Definitions & Players article to understand why what you think is real time might not be — unless you disagree with my view of what real time “content” is. In that case, leave a comment!

Find Books, News & Blogs

Up at the top of the Search Options column links that allow you to narrow results to these areas:

• Videos
• News (Added today)
• Books (Added last week)
• Reviews
• Forums
• Blogs (Being added today)

The ones in bold are new. Books was added quietly last week. News was added today. Blog is supposed to be added today, so keep watching, if you don’t see it.

Forum Sitelinks & Trends OneBox

Along with the new Search Options, Google’s also reminding people about the new forum listings that it added yesterday (see Google Rolls Out Sitelinks Display For Forums) and the Hot Trends OneBox unit that’s been added (see Take That, Twitter: Google Hot Trends Integrated Into Google Search).

And will there be more to come? You betcha, says Google.

“Usage [of search options] has been strong and increasing more and more,” said Nundu Janakiram, an associate product manager at Google, about the changes. “The steady increase in using the tool tells us we should try to add new and interesting features into the panel.”

What percentage of people use the panel? Revealing that data would cause the Earth’s rotation to stop, or the universe to collapse into a black hole, or — yeah, Google’s absurdly paranoid and won’t say. But I’m hoping they’ll loosen up in the near future. But they would say that among the options, the date and time filtering ones are the most popular.

Hey, that Up Close With Google Search Options article that I mentioned? Really consider reading it. It covers the things above in way more depth.

Now that Google Search Options has added new features, I thought it was a good time to revisit how everything works — and in some cases, doesn’t work. Let’s get up close and personal with all the filtering options!

By the way, this is a long article. If you want a digest of what’s new today, then see the much shorter companion piece, Google Adds Visited Pages, Past Hour & Fewer Shopping Sites Filtering.

Using Search Options

The Search Options panel is available after you do a search. Look near the top left-hand side of the search page, just below the search box, and you’ll see a “+ Show Options” link:

After you do this, a series of options will appear to the left of your search results:

Selecting any of these will cause the results to change, based on the filter you choose. Not shown in the illustration above are the Viewed / Not Viewed choices, but those will be explained (and illustrated) below. This article will run from the top of the Search Options panel to the bottom.

Filter By Result Type

The first set of choices allows you to filter the results to show specific type of content:

1. Videos
2. Reviews
3. Forums
4. Books (New)
5. News (New)
6. Blogs (New)

Video Searching

Video filtering was one of the first three filters available when Search Options launched. Select this, and the standard search results are filtered so that only video clips show:

Note that when you do this, as the arrow shows above, you get new “Any duration” filter options that let you further refine the video results to show clips that are short (0-4 minutes), medium (4-20 minutes) and long (more than 20 minutes) in length.

Video Searching Confusion

Like getting video results? Confusingly, the Search Options feature means you now have two different ways to get them on Google, which in turn produce different experiences.

Look at the very top of the search results page, and you’ll see a Videos link in the navigation bar (the top arrow in the screenshot below shows this). This was rolled out in May 2007 with great fanfare (see Google’s New Navigational Links: An Illustrated Guide). Click on that Videos link, and you get results back from Google Videos:

You’d think that for consistency, getting video results by using the top-of-the-page navigational link should bring back exactly the same thing you’d get by filtering for videos using the search options panel. But not so, as I’ve highlighted in the screenshot above.

For one, the ranking of results is different. Using the navigational video link, I got a result on replacing an iPod battery that did not rank in the top results when getting video using Search Options. Also, using the navigational link puts the first video results within a blue border, making it seem like an ad, even though it’s not.

More confusion. If you search for videos using the navigational link, you can still open up the Search Options panel — whereupon you find options that are NOT available if you do a regular search and then filter by video. In particular, the navigational link enables different display options (TV view, list view & grid view), along with options to see only high quality video, to filter by video source site (such as YouTube or Vimeo), to see videos with closed-captioning and more.

There are even more filtering options available for video searching, such as to filter by language or file type. However, to get these, you have to use the advanced search page at Google Videos.

Review Searching

Another filter available at launch is the ability to get back what are supposed to be product reviews. I used the word “supposed” because despite the promise, you’ll still get occasional “review” that can be puzzling. For example, here are “review” results for ipod:

Many of the results lead to good review sites (I’ve marked these “Good!” in the screenshot above). But down there at the bottom of the page? Yes, the web site that will not die — Wikipedia. I always joke that it’s required by law for Wikipedia to be on every Google search results page. Perhaps it really is a law.

I can already hear Google spam czar and all around debunker Matt Cutts winding up a defense. “If you look at the Wikipedia page listed, you’ll see there’s a criticism area and a useful history of the models.” True, but it’s still not what I’d expect when I’m thinking product review.

Neither am I thinking that the manufacturer of a product — in this case Apple — should be listed (twice). I went to both of those pages, by the way. Neither had product reviews.

In the middle of the page, you get a big huge shopping results OneBox unit. Hey, I thought I was getting review listings, not shopping listings!

As it turns out, those shopping results do have a nice collection of reviews, like you’ll see here, gathered from across the web. In fact, those listings seem better than some of the “review” listings that search options was giving me. So why aren’t these pages listed like “normal” pages rather than being confusingly tucked in a shopping OneBox? Why isn’t Google perhaps showing a better display of these (mostly user reviews) somehow mixed or set alongside editorial reviews?

Forum Searching

Forum searching was the third filter available at launch. It works well. You do indeed get back matches from online forums and other places where discussions are happening. Still, I’ve got gripes:

Note that the VMware Communities are both the second and the third major listings. Sandwiched between them are three more VMware Communities threads using the new forum sitelinks display. That’s five listings from the same site in all, showing up in three different places. Surely there could be more consistency here.

Book Searching

Last week, Google quietly added a book filter to Search Options. Select this, and you get back matching results for your search from Google Book Search:

Note that after filtering to matching books, you also get new options to see “Full View” books (that you can read fully online) or to show either books or magazines. The arrows in the screenshot above point to these.

The ranking of results seems to match that as if you went to Google Book Search itself. What’s missing are the many more filtering options that Google provides through Google Book Search’s advanced book search page (such as by author or by ISBN).

News Searching

The news search filter was added yesterday. Select it, and you get back matching news results from Google News Search. The ranking and display is identical, as best I can tell, to what you’d get at Google News Search itself. The main difference is that if you search at the dedicated news site, the Search Options panel changes to provide additional date filtering options and the ability to filter to just news images:

See the year blocks, like 2008-2009 or 1980-1989? You only see these when using search options via Google News.

In either case, there are even more filtering options such as news source or author which only appear if you use Google News Search’s advanced search page.

Blog Searching

Blog filtering is supposed to be added sometime today. As it wasn’t live when I wrote this article, I couldn’t test it. I’d expect it to bring back results from Google Blog Search and for the Search Options panel to reflect some, but not all, of the filtering options at Google Blog Search’s advanced search page.

Date Filtering

Filtering search results by date is hardly a Google innovation. Many search engines offered this in the past. However, it never really seemed to catch on. My personal theory was that most of the time when people are wanting to filter by date, they want to have “recent” results — which means news results. In turn, that means they should be using a news search engine.

Another complication is that knowing the “date” of a page has long been a messy business. Is the date:

• When the page was first published as reported by the web server (sometimes these give out incorrect dates)
• When the page was updated with new material? (such as an article that was written, then perhaps revised months or years later)
• When the page was first found by Google? (which might not mean the page was actually published on or near that date)
• The “date” that might be listed somewhere on a page, such as near the author’s name?
• The “offline” date (such as if an old public domain article from the 1700s is put online)

In most cases, Google tells me the date will be when it first visited the page, though it will try to combine various signals to come up with the best one to determine a valid date.

Date Inconsistency

That sounds good, but it’s pretty easy to demonstrate how messed up the date identification can really be. For example:

The first listing says “Sept. 15, 2009″ but in reality, that page has been on our Search Engine Land site at that exact URL for nearly a year. So, the date isn’t the “first visited” date. If you go to the page, you’ll also see that it has content as of Sept 25, 2009 — so that isn’t a “last updated” date being shown. As for the web server, it spits out a date of October 1, presumably because the page comes from a database. Each time it’s requested, that’s the “date” of the page as far as the server is concerned.

This all means the date Google shows is the date of the last time its spider visited the page. If you view the cached copy of the page, you’ll see the date there is also Sept. 15. However, that’s misleading. The dates on Google’s cached pages can be days, weeks or even months out of sync with when Google last spidered a page to update its searchable index.

The second listing seems to use the date as published on the web page, the date displayed to readers, in order to inform them of when the page was written. But then the third listing ignores that and like the fourth listing, seems to use the last visited date. Then the authored date gets used again, then for no apparent reason, the authored date gets ignored and the last visited date appears.

Date Options

My Finding Search Engine Freshness & Crawl Dates is an older article that goes into depth about issues with dates, if you really want to know more. But despite the fact that dates might not always be accurate, I’ve personally used the filter by date option many times to successfully narrow down results in a useful way. Nor am I alone. The date and time options, Google tells me, are among the most used from those offered in the Search Options panel.

The date options are mostly self-explanatory. When Search Options was launched, these were offered:

• Past 24 hours (results with a date in the past 24 hours)
• Past week (results dated in the past week)
• Past year (results dated in the past year)
• Recent Results (undefined — in one test I did, results stretches back to the “recent” year of 2002. I’m checking on this)

At the end of July, a custom date range option was added, allowing for pages to be narrowed down between a specific period of time.

Today, a past hour option was added, allowing you to find pages dated within the past hour. That not good enough for real time search junkies? Last month, the Omgili Blog discovered a way to narrow results down to the last minute or even the last second, if you’re willing to play around with the URL that shows up after a search. It’s easy and safe. Our Hidden Google Feature: Find What’s New In The Last Minute Or Second covers it in more depth. For the record, Google confirms they work and simply calls them unsupported.

I joked about real time search junkies, but in seriousness, I have a severe allergic reaction to anyone who believes that Google finding a page in the last minute or second means it provides real time search. For me, real time search means finding a particular type of “real time” content. My What Is Real Time Search? Definitions & Players article goes into depth about this.

Finally, let’s have a look at all the date options in context:

The top arrow points to the narrowing options you have. But once you’ve selected these, the bottom arrow points to a sorting option. By default, results are sorted by relevancy. However, you can sort by date within the range you’ve filtered. Do that, and the most recent comes first. There’s no oldest to newest option, which would be useful, on occasion.

Visited / Not Visited Pages

To me, the new Visited / Not Visited Pages filter introduced today seems like a really useful feature. Assuming you use Google’s Web History feature, you can have Google filter out pages you’ve already clicked on from its results or feature those pages and hide those you’ve not seen.

For example, I was looking for information on how to add Twitter to Google Wave earlier today. I’d heard about a program called Twave, so started searching for it. By using the “Visited Pages” option (in the screenshot below, look in the middle of the left column), it was easy to get a list of what I’d been to:

Notice the two arrows to the top right of the screenshot. They point out how Google tells me when I last visited the page listed (me visiting the page, not Google’s spider) and the term I searched for when I clicked from Google’s results through to the page.

All’s not perfect, however. Notice the three arrows further down. Technically, I told Google to do a search for “twave” and then filter out only pages that I’d visited when doing that particular search in the past. So why’s Daggle, my personal blog, showing up? Look at the bottom two arrows. They show I last visited the page on September 12 for a search on “daggle email,” not on “twave.”

The answer is shown in the description. See how I’ve boxed and pointed at the word “Twave.” I wrote a post about Twave on my blog. Google visited my blog, saw that word and so considers it a page I’ve visited before for a search on “twave” simply because that word is on the page — NOT because I actually searched for “twave” in Google and found my blog that way.

I especially liked the Not Visited option:

I can think of so many times I’ve done a search, then realized I was “done” with some of the pages I’d already seen but kept getting them back when I searched again. That was the case this evening, when the pages I viewed didn’t really have the answer I wanted. I need to look at some more.

Unfortunately, most of the other ones I hadn’t visited because I could already tell they were junk by looking at them. Yet the only way to remove them from the Not Visited list is to actually visit them? Ugh.

An easier solution would be if Google enabled its SearchWiki feature when you’re using Search Options. Then you could easily delete pages you weren’t interested in. For some reason, SearchWiki isn’t active when Search Options are used (for more about the service, see my Google SearchWiki 101: An Illustrated Guide article).

Finally, you’ll only see the Visited / Not Visited pages option if you’re logged into Google and making use of its Web History feature. That will have an ick factor for some people, the idea that Google’s watching what you search for and click on. If you’re among them, well, don’t sign in and don’t use Web History. All the other search options will work just fine for you. My Google Search History Expands, Becomes Web History story goes into more depth about the Web History feature. Like really, really in depth. It makes this article seem short.

View Options

Further down in the Search Options panel are four ways to view your search results, all of which were present when Search Options launched:

• Standard view
• Related searches
• Wonder wheel
• Timeline

I’m not going to drop a ton of screenshots and explanations about how these work, because I’ve already done that in my Google Wonder Wheel & Other Search Refinement Features article. Check that out. The only difference is in that article, the “Related searches” option was called “Search suggestions.”

Results Options

Just below the view options area are results options, like this:

It’s confusing, I know. Didn’t we have a filter by results area at the top of the Search Options section? Yes, we did. And isn’t “More Text” a view option. Yes, it is.

The “More text” and “Images from the page” options were present when Search Options was launched. They show longer description for results and thumbnail images alongside results, respectively. That Google Wonder Wheel & Other Search Refinement Features article I mentioned earlier explains more about the options, complete with illustrations.

If it were me, I’d move both of these options into the View Options section. As for the other two related to shopping, I’d put them into that other results options area — the one where you can get news, video, blog results and so on.

Let’s talk about those shopping optons. They’re brand new: “Fewer shopping sites” and “More shopping sites.” I’m sure they’ll also freak out some site owners who’ve long suspected that Google’s wanted to decommercialize (if that’s even a word) its results in order to push businesses into buying ads. And in fact, to some degree Google admitted that type of shift back during the major “Florida Update” of 2003. Not to boost ad sales, of course, but because sometimes people may want less commercial results. Well, now you can overtly drop shopping-oriented sites from your results or conversely, pump them up.

Here are standard results for rollerblade speedmachine, which as you can see are loaded with shopping results:

When I apply the “fewer shopping sites” option, however:

Away go some of the sites, with the boxes showing where new forum threads or blog posts have come in. Shopping results are still there, though kind of sad, I noticed a good dependable store I use personally disappeared. I’d like the feature better if it removed shopping sites with less reputation, if it’s going to leave some in. But I’ve also not done more than a few tests, so perhaps this isn’t the case with other searches.

Wondering what makes a site shopping-like? Google says prices are one of the key signals. You have a lot of prices, you may seem like a shopping store. Google also said things that just look and feel like shopping sites will get flagged. If your using certain words frequently associated with shopping sites, or a format that is commonly seen, that can also be an influence.

Conclusion

Don’t see some or all of the new options mentioned? Remember that for the Visited / Not Visited options, you need to be logged in. As for others, typically with these types of releases, it might take a few days for everyone on Google to see them.

Overall, I like that the new options are more visible, and it may perhaps be reversing the long standing conventional wisdom that searchers simply ignore options when offered through “advanced search” or other links. However, it feels like Google’s been so busy growing the Search Options panel that it’s failing to maintain consistency with the existing advanced search pages for some of its vertical search properties. I’d like to see them consolidate and be more consistent.

Google’s “normal” listings show the title of a web page, a description of it (also called a “snippet”) and the web page’s URL. However, Google also has other listings that appear within search results that are designed to give access to some of its specialized search tools (such as news search), to allow more information to be shown than a standard answer provides or to show answers directly within the search page. Here’s a short overview to these alternatives.

OneBox Results

OneBox results are when Google shows information within a special unit, often with images associated with them. OneBox unit often appears to highlight news, shopping, image and other results that are blended into regular listings using Universal Search (see Google Universal Search for more about this)

In the example below, you can see three different OneBox units all mixed among regular results:

OneBox, by the way, gets its name from the idea that Google searchers should be able to use one single search box to access information from the various Google specialized search engines that exist.

Local OneBox Results — The “10-Pack”

For local results, Google does a special OneBox display that has a map on the left side of the OneBox and 10 local listings to the right. Here’s an example:

Those who track local search closely call this the “10-Pack” or “Ten Pack” display, because of the 10 local listings that are packed into such a small area.

Plus Box

Google “Plus Box” results are where Google shows a search listing with a little + symbol nearby that when selected “expands” the listing to show more information. For example, in the search on “dell” below, you can see a line that says “+ Show stock quote for DELL” like this:

Clicking on the + symbol displays a stock price chart and further information, like this:

Direct Answers

For some popular queries, Google tries to show an exact answer to what you’re looking for right at the top of the search pages. For example, here’s a search on weather in Newport Beach:

See how the weather report is shown directly in the results? That’s a direct answer, and Google does this for everything from sports scores to flight tracking. Here’s a list from Google of just some examples of direct answers you can try.

For more about the features shown above, see the Google: OneBox, Plus Box & Direct Answers section of the Search Engine Land Library. Also see the Google: Search Customization and Google: User Interface sections for other ways Google customizes its search results.

As if SEOs don’t have enough things to worry about already, add malware to the list. Why does malware matter to SEOs? If the site you are working on gets infected, its search traffic will plummet. Search engines attempt to remove infected pages from their search results, or they label them with an ominous warning, such as This site may damage your computer.

Back in 2008 Google reported that malware infected pages had increased to more than 1% of all search results. Google posted a malware statistics update last week. Malware infections have more than doubled since April 2009. Search results containing a url labeled as harmful have remained level in the range of 0.5% to 0.9%, an improvement. While the web as a whole has become more dangerous, Google’s been doing an even better job clearing their search results.

I know one reason why there’s been a dramatic rise in malware on the Web since April. A nasty malware attack has been targeting web developers to steal their passwords. Stolen passwords are used by the bad guys to automatically deploy iframe injection attacks to innocent web page.

If you access web sites via File Transfer Protocol (FTP) or Secure File Transfer Protocol (SFTP), this attack is targeting you. All you need to do is browse an infected page using an insecure browser. Badware will be deployed to your machine, and it will find the files used by FileZilla, or possibly other FTP programs to store passwords, and silently send those files back to a server in China. Then an automated bot attack will use FTP to edit your web pages, infecting them with malware. Then your sites will drop out of the search results. Can you image the uncomfortable conversations when all your sites get hacked at once and you have to admit responsibility?

What can be done to reduce this risk of search Armageddon?

1. Use a more secure browser such as Chrome or Firefox with the NoScript add on for routine browsing.
2. Don’t use any FTP program that stores passwords locally in plaintext, such as FileZilla. To date, Dreamweaver has not been reported to have been compromised. Dreamweaver encrypts passwords and stores them in the Windows registry.
3. Consider using a Mac or Linux instead of Windows. As the most popular operating system, Windows is the most popular target for attacks.
4. Make sure your machine and server are fully updated and patched. Turn off unnecessary services and software to reduce the attack surface.
5. Register your site with Google Webmaster Tools and Bing Webmaster Center. Check regularly to see if there are any malware reports (or other issues) with your sites.
6. If you suspect a malware infection, check Unmask Parasites,
7. View your site’s reputation at McAfee SiteAdvisor.
8. Reduce the number of people and computers that have access to your web server.
9. Keep a backup copy of your web pages. In case of infection, it’s a race to see if you can fix the site before search engines (and users) discover the problem and dump you.
10. Choose the hosting provider that has the quickest response time, not the cheapest price. If your site gets hacked, you may need their help to change all the passwords.

As the web becomes more dangerous, customers become more suspicious, reducing opportunities for everyone. Please do your part to make the web safer, and to reduce your risks.

I wrote earlier that it was probably driving Google crazy to see people “amazed” that Bing is doing stuff that Google itself has long offered. Now Google’s fighting back. The first shot? A new Explore Google Search page.

Google’s had help pages like this before. For example, this one still remains, showing various special features such as the ability to get weather results. But this new page is being featured right from the Google home page:

The page frames many of Google’s features in an easier-to-read format:

And it comes with an introduction highlighting how serious Google is about search:

Search is at the heart of everything we do at Google. Our engineers work every day to solve the hardest search problems, and thus improve your online experience. Here’s a glimpse at what they do, the features they’ve built and the remarkable user stories that inspire our work.

As I said before:

Google’s never really had to market itself to consumers, to trot its stuff. The recently held “Searchology” event didn’t cover anywhere near the range of what Google offers. But if the praise for Bing keeps largely rolling in — if people keep discovering features that aren’t necessarily unique to Bing — Google may find it has to step up.

To me, this page is all about doing that stepping up. Of course, the page is well worth reviewing, for those who aren’t aware how Google can provide everything from sports scores to flight tracking. As can Bing, and as can Yahoo.

Yahoo! recently announced their role in creating and supporting Common Tag, a new semantic tagging format.

Yahoo! says that Common Tag makes “web content more discoverable” and enables the community to “create more useful applications for aggregating, searching, and browsing the web.” Their blog post mentions that they want to accelerate the structuring of the web, which aligns with their SearchMonkey launch last year, which they said was, in part, an attempt to encourage the use of structured data on the web. This brings to mind a few questions.

Why did the web need a new semantic standard? The Common Tag blog explains:

“Semantic web promises havens. The promise of machines understanding the data and acting on it semi-autonomously. But how do we get there? Not many are willing to put in a lot of work into publishing data in new formats. For the benefit that might or might not come years after enormous resources will be spent?

So why not start with something easy, say tagging? Marking up text with exactly defined concepts. “

OK, maybe “explains” isn’t the right word.

This question really goes to the heart of what’s curious about Common Tag. Yahoo called it a “new semantic tagging format” in its blog post, but when we asked them why the web needed something new, they clarified that it’s an RDFa vocabulary, not something made from whole cloth. The Common Tag About page also implies that this is really just part of the standards that all the major search engines have joined together to support. “In addition, search engines like Yahoo and Google have begun reading RDFa—the markup standard used by the Common Tag format—to acquire richer information about sites that use it… Google’s new Rich Snippets feature uses the information to apply similar enhancements to Google search results.”

In truth, none of the major search engines are using semantic markup in web search and Google is using existing standards (microformats and RDFa) to display enhanced listings. Both Google and Yahoo have told me that they could use metadata in web search in the future, if it proves to be useful and they can safeguard against spamming. So far, this hasn’t happened.

Yahoo did clarify to me that Common Tag is something they’re participating in as means to cultivate the structured data community, not something they’ve come up on their own and are trying to get the community to adopt. RDFa provides a structure from which you can create vocabularies and several companies who were using RDFa were interested in creating a tagging vocabulary. Since these companies used SearchMonkey as an application for their metadata, they asked Yahoo to help create and promote  this new vocabulary.

So, how does it work? Common Tag is intended to be a common tagging format to standardize tagging of concepts. According to the commontag.org site, as “publishers, developers, and end users” join in support for this format, “more content related to a specific concept will be discoverable through a single tag.” Now, for instance, the concept New York City may be tagged with “nyc”, “new_york_city”, and “newyork”. You can adding this tagging markup to your pages manually, or you can use infrastructure such as that provided by founding company Zemanta. And you can eliminate the problem of multiple tags for the same concept by using data from a participating database, such as Freebase (also a founding company). You can then use this structured data in an application such as Yahoo! SearchMonkey.

For instance, the Common Tag documentation uses the following example of using the Freebase database to tag a page as being about U2:

<body xmlns:ctag="http://commontag.org/ns#" rel="ctag:tagged">
    <span typeof="ctag:Tag" rel="ctag:means"
         resource="http://rdf.freebase.com/ns/en.u2"/>
</body>

You can also do more complicated tagging, such as of external resources, sections of your web pages, and concepts within your content. For instance, you can identify the paragraphs of text on the page as follows:

<p id="first">Everyone loves Buffy the Vampire Slayer. </p>
<p id="second">Amber Benson was awesome in it.  </p>

And then create tags for those paragraphs:

<div xmlns:ctag="http://commontag.org/ns#"
         about="#second" rel="ctag:tagged">
    <span typeof="ctag:Tag"
         rel="ctag:means"
         resource="http://dbpedia.org/resource/Amber_Benson"/>
</div>

Why is Yahoo! so hell-bent on covering the web with structure? If Yahoo! found structured data made the web easier to crawl and their search results more relevant, I could see the push. But Yahoo! doesn’t use any of the semantic formats they’re encouraging in web search. They already were encouraging hCard, hCalendar, hReview, hAtom, XFN, Dublin Core, Creative Commons, FOAF, GeoRSS, MediaRSS, RDFa, and OpenSearch. Why do they need web developers to start using yet another format when they haven’t yet figured out how to use all of those others in their core search engine? Sure, they are involved in Common Tag in order to support the structured data community they’ve been aiming to accelerate, but why is that so important to them?

Since Yahoo isn’t encouraging the use of semantic markup to help them get an edge in search, it seems they must be instead looking to increase adoption of SearchMonkey and BOSS, where these formats are used.They seemingly have diverted the energy they used to spend to help improve Yahoo’s search index via tools such as Site Explorer to working to raise adoption of BOSS. The last Site Explorer update was in August 2008, and that was simply a UI change. No new features were launched. For new features, you have to go all the way back to August 2007, for dynamic URL rewriting.

Even Yahoo CEO Carol Bartz isn’t talking about focusing their consumer search engine as a core offering, but rather something that’s convenient for Yahoo users who are already on the site for some other reason (emphasis mine):

Listen, Google has the search brand, there’s no doubt about it …. [Yahoo Search] is really designed for people that are on our sites and find something interesting, they want to look farther and they go to Yahoo Search.

It’s enough to make one wonder if Yahoo is quietly abandoning its consumer search engine in favor of accelerating new third-party search engines through BOSS. If you can’t beat ‘em, help their enemies attack them on all fronts, as the old saying goes.

Didn’t the search engines already try using meta tags? The idea of using meta data to tag web pages in order to describe them to search engines isn’t new, of course. The meta keywords tag has been around since at least 1995. And it’s easier to adopt than Common Tag. That U2 example? The meta keywords tag would only require this:

<meta name="keywords" content="U2">

Indeed, Yahoo supported the meta keywords tag initially (and to some extent, still does), but when Google launched, they did not. It was too easy for site owners to stuff that tag with anything they wanted, rather than the true focus of the page. Search engines use smarter methods (starting with the content on the page and how external sites link to it) for determining relevance. Could Common Tag have the same downfall? After all, as the documentation explains “you can create as many Tags as necessary to describe the contents of a document.”

Not only does Common Tag seem to replicate the purpose of the meta keywords tag, it seems to also replicate Delicious-style tagging and external anchor text. From the site:

“Common Tags are not only useful for identifying the concepts covered in your content, but if you reference content elsewhere on the web, Common Tags can be used to indicate the concepts covered in that external content as well. This is useful for better describing and organizing the content of external resources from within your own content. For example, you could use Common Tags to publish bookmarks to identify the concepts described by a link, or you could use them categorize image collections stored elsewhere on the web.”

A microformat already exists for a similar purpose as well.  rel=”tag” is intended to tag content, such as web pages or portions of them.

Anchor text is an established method for search engines to determine how others describe an external resource. As for tags, the study Can Social Bookmarking Improve Web Search, presented at the First ACM International Conference on Web Search and Data Mining (Stanford) analyzed 40 million Delicious tags and found that anchor text was a better signal for web search relevance. Part of the problem was scale of adoption. A lot of people have to adopt this new tagging method for it to be worthwhile to use across the web. And if Delicious tags don’t have the scale, how long will it take for Common Tag to?

When I asked Yahoo about it this, they acknowledge that it may not be something that’s adopted web-wide. Rather, it’s a format of interest to a particular group of developers who have needs beyond that which is available through means such as the meta keywords tag and rel=”tag”.

Why would anyone implement this? It seems like a lot of work. You can tag content now using methods like anchor text and well, tags, such as those available through most blogging platforms and bookmarking sites like Delicious. If content management systems and other content creation platforms such as blogging systems incorporate this structure (for instance, by automatically using the tags labeling a blog post), we might see some adoption, but this wouldn’t eliminate the issue of multiple tags for one concept. (Zemanta, one of the founding companies for Common Tag provides plugins for blogging platforms to insert Common Tagging.) And Wordpress strips out RDFa by default.

The answer is that web developers will use this structure, just as they’ll use any other structure, if it’s valuable for what they’re building. And what applications are ultimately made possible by this format remain to be seen.

The Microsoft Translator Blog announced that translation is now an instant answer in Bing, Microsoft’s new search engine.

Here are some examples:

Google added limited translation as a OneBox over a year ago. But it seems like both Yahoo and Ask.com do not have smart answers for translation.

Hat tip to LiveSide for spotting this.

MobileCrunch noticed Yahoo has added voice search capabilities to their Yahoo One Search iPhone App. I spent a few minutes testing it out and I have to say, that is does an okay job. I didn’t conduct enough tests to really compare it against Google’s iPhone voice search feature. They are both different in how they work.

First let me show you Yahoo’s voice search for a test on [who is barry schwartz]:

Step 1: Click on the “Press + Speak” button to activate voice search:

Step 2: Speak when button turns red and then tap it again when you are done talking:

Step 3: The button turns blue to show it is processing your search:

Step 4: Yahoo shows the search result, and in this case, wrong (I tried three times):

Google’s voice search works a bit differently.

Step 1: Either put the phone to your ear or click the microphone icon:

Step 2: Just speak and when you are done speaking, Google knows and shows that it is processing your search:

Step 3: Google displays search results:

So, Google is one step less, which is nice for a mobile search app.

Google has announced several changes that are coming soon to Google Suggest, the feature that suggests search queries as you type.

The two most interesting to me are the inclusion of ads and direct links in the search suggestions. Let’s take a look, starting with the suggested links.

Navigation Links

If Google is confident about what you’re looking for, it may turn the first suggestion into a link. The example Google cites is a link to the official Kennedy Space Center web site as someone types “kennedy space cen.”

Users who already have access to the new feature have had fun trying to see how few characters they can type to get one of these navigational links; three seems to be the minimum found so far. Oh, and if you’re thinking there might be some favoritism at play in what links are tied to which queries — yes, typing “gma” brings up a link to Gmail, but typing “map” brings up a link to MapQuest, not Google Maps.

Ad Suggestions

In some cases, Google may show an ad at the bottom of the suggestions. And for now, ads will only show from advertisers that Google has whitelisted.

Google makes an interesting choice of words in its announcement to describe when this might happen: “…sometimes we detect that the most relevant completion for what you’re typing is an ad.” It’ll be interesting to see how often ads show up in search suggestions; we (and others) have noticed that Google has been placing ads anywhere and everywhere for the past several months.

Other changes announced today include:

• In addition to the home page, suggestions will be offered from search results pages, too, and the first few suggestions may be related to your most recent search.
• Suggestions may tailored to your search history (if you’re logged in and have Web History enabled).
• Result counts won’t be shown anymore.

Google says these changes will be rolled out gradually. There’s more discussion on Techmeme.