It’s the 20th anniversary of the fall of the Berlin Wall. Where’s the special Google logo for that, some have wondered. There is one, but only at Google Germany. Elsewhere, it’s apparently more important to keep celebrating the 40th anniversary of Sesame Street.

The Google Germany logo is shown above. At Google’s main Google.com site (which serves the US), it’s Count von Count from Sesame Street:

You’ll also see Count von Count in places like Google France and Google UK but not at Google Russia (it gets neither Sesame Street nor a Berlin Wall logo). All three countries, along with the US, had post WW-II “sectors” of control over Berlin.

I’ve actually been working on a piece about how Google determines when to run logos and for what topics. That’s not ready to go out yet, but suffice to say, there’s not a lot of rhyme or reason to it. Sometimes, the company stumbles into offending someone. A national holiday gets overlooked or not marked each year, for example.

In this case, I think it’s fair to say that while the Sesame Street logos were creative and fun, the seven day saga was probably overkill — and someone should have been checking dates for conflicts with other important events.

By the way, Bing has the same oversight. Its US home page features a Tyrannosaurus Rex fossil, as does Bing France. Bing UK has a picture from what’s apparently a viewing gallery inside the German parliament building:

Bing Germany goes with a classic East German Trabant car, set in front of the Brandenburg Gate, which used to be surrounded by the Berlin Wall and was a focal point for celebrations when it fell:

Over at Ask, the US site has a big picture of people in front of the wall:

The UK site has Ask Jeeves (yes, Jeeves is back there) wearing a Remembrance Day poppy (these are commonly worn in the lead-up to the actual day to remember those fallen during wars, which is on November 11):

The Ask Germany site has nothing. And speaking of nothing, that’s what you get at Yahoo, either at the main Yahoo.com site, in the UK or Yahoo Germany.

Like many people, I misspelled a domain name today when I was trying to visit a web site. I typed Twiter.com (with one T) rather than Twitter.com. I wasn’t surprised to land on a site with ads, as is common when entering typos. I was surprised that both Google and Yahoo were making money off those ads.

Google has a program known as AdSense For Domains, previously known as DomainPark. Got a tasty domain but no content for it? AdSense For Domains will put lucrative ads up on it, for you (really lucrative: see more here and here).

The practice is known as domaining. And before some domainers start working up heated rebuttals, let me make it clear. There’s nothing wrong with domaining. If you were lucky enough or smart enough to land a generic domain like usedcars.com or taxforms.com, my hat’s off to you. It’s well known that people will simply slap words together, tack on a .com and see if they reach a site that has information about a particular topic relating to those words. Domainers earn off that traffic, and no one is misled when visitors directly navigate this way.

So saying domaining = spamming is the same as saying SEO = spamming. Neither is true. But there are spam tactics that happen in both areas along with the legit stuff. In the domaining world, it’s the typo traffic that’s often scummy, in my book.

Typo domains are domains that are nearly identical to the domain name of another well known brand. Here, there is often harm. Someone expecting to reach a particular site instead lands on a different one that’s cashing in on the other brand’s fame.

OK, it’s the person who is entering the domain name wrong in the first place’s fault, right? It’s like they dialed the wrong phone number. Why shouldn’t a domain owner be able to earn off of misdirected calls to their phone? Or, it’s the “real” company’s fault for not registering all the typos out there.

What about companies that have a name in use before another company becomes more famous? Is there really anything wrong with UTube — a well established pipe company — benefiting from a spike in traffic after some upstart YouTube video site came along? Or in the case of Twiter, that domain existed well before Twitter became popular, so why shouldn’t it tap into new found popularity.

These are fair objections. In counter to them, some typo domains are often registered after a brand becomes popular, with the obvious intent of riding on someone else’s coattails. For another, it simply violates the policies of some ad networks, Google’s included. In other words, the fault isn’t with the domain owner themselves. It’s with companies supplying ads in violation of their own guidelines or policies.

That leads us to what I saw when I reached Twiter.com, the single “T” web site:

These ads are provided by Google, not that anything on the page tells you this. Domain ads apparently aren’t forced to carry those “Ads By Google” notifications as with contexual ads. That’s a handy way for Google to distance itself.

The first and fifth ad indicates a relevancy issue for Google advertisers. If you’re advertising “Free VoiceXML platform” or “Monitor Server health,” why on earth is someone from Twiter (one T) going to convert for you? They might click out of curiousity, but the probably aren’t going to buy (in fact, there’s a lawsuit against Google over the quality of domain ads pending. Google’s has also been sued over trademark issues with typo domains).

Now look at the second ad, which I’ve pointed an arrow at:

Twitter? Twitter is here Need Twitter? Official Twitter site Twitter lets you share. Its Twitter www.Twitter.com

That ad surprised me. Really, Twitter (that of 2 Ts) decided to buy an ad for its own name via Google? Actually, no. Instead, it’s a Twitter user that bought the ad, driving people to their particular profile which, while indeed being on the official Twitter site, isn’t providing Twitter but rather a pitch for a book:

Clever person, right? Yes, but they also likely being misleading. That would violate Google’s ad guidelines and also may violate advertising laws in various US states, as well as nationally and in other countries.

That ad also shows two flaws in Google’s ad system. Clearly no human being looked closely at this ad, to review it for quality guideline violations. Meanwhile, Google’s requirement that the display URL in an ad match the domain name someone arrives at get exploited. This ad correctly shows a Twitter.com domain, even though the ad itself doesn’t speak with the authority of Twitter itself.

Check out the third ad, with an arrow pointing at the domain (which I’ve also bolded below):

Twitter Looking for Twitter? Find exactly what you want today. Yahoo.com

Why yes, I was indeed looking for Twitter. Glad to know that Yahoo has it now. I guess I missed news of that deal being cut. Let’s go get us some Twitter at Yahoo:

Ah, Twitter, er, shopping results. Maybe that lamp beams out tweets, when you turn it on. The results are kind of crummy. But that’s OK, because right at the top of the page, we get three paid search ads from Yahoo.

Now, it’s not Yahoo doing this directly. Looking at the URL that brings me to the shopping page, I see an affiliate reference. So this is someone earning money by driving Yahoo traffic. But Yahoo takes some of the blame here. It’s their affiliate, getting paid by Yahoo, and Yahoo should be policing this.

Yahoo, by the way, joined a coalition against typo domains back in 2007. They’re no longer listed as a member, which given these type of ads, is probably best.

The rest of the ads are all products somehow related to Twitter, so at least the misleading aspects aren’t there. But there still seems to be a violation of Google’s domain ads program policies:

Domains submitted for the AdSense for domains program may not violate any trademark (and related rights), copyright, trade secret, patent or other intellectual property right of any third party….

Google AdSense for domains is committed to respecting the rights of trademark owners. It is our goal that advertisers, users and trademark owners all be aware of Google’s process for reviewing perceived trademark infringement in the AdSense for domains network. If Google becomes aware of a domain name that contains a trademark (or typo), that domain will be removed from the AdSense for domains network.

At best, Google might argue that Twitter hasn’t submitted a formal complaint, so as far as it knows, there’s no trademark violation happening. That’s still pretty weak. Does Google, which often holds itself out as championing the relevant organization of information, really want to hold its head up about what’s happening on that page?

I don’t think so. That’s especially so when you consider the type of ads that show up on Google’s own site for a search on twitter:

That Twitter user claiming to be the official Twitter site doesn’t show there. Neither does the Yahoo ad promising to deliver Twitter.

If those ads aren’t good enough to be shown on the shining storefront that is Google’s search results page, they don’t get any better being plastered on some dark alley of the internet.

Postscript: After publishing this, I sent these questions to Google.

• Are the ads from that Twitter user and from Yahoo meeting your relevancy guidelines?
• Are they not misleading?
• If they are, were these actually reviewed by a human?
• And does the site violate your guidelines on typo domains or not?

In response, I was emailed this statement:

We don’t comment on specific ads or domains – but our AdSense for Domains policies are here. When we’re notified of complaints, we investigate for compliance with our policy.  We’ve found that advertisers enjoy the benefits of the additional reach that AdSense for Domains offers.  Many advertisers find that ads on parked domains perform as well as or better than ads on more traditional search and content sites.

Today, the site is no longer showing ads from Google. Instead, another company is providing the paid ads.

Google’s now officially announced “Comparison Ads,” starting with a way for mortgage advertisers to have their products compared against each other but which may expand to other products in the future.

“If you’re looking for a mortgage, you really care about a specific offer. You want to know the APR and based on your specific circumstances, which is something that AdWords doesn’t provide today,” said Nick Fox, director of business product management for AdWords at Google.

In the new system, those search for “mortgage” or “refinance” or related terms may see an special ad from Google itself inviting them to view offers, like this:

Note that while the product was just announced, not everyone may see these ads yet. Google says they’ll be fully deployed over the next week. They’ll be visible in specific states where the ads have matching content (more about that below).

Clicking on that ad takes them to a page where they can view a variety of related mortgage offers from various lenders, like this:

You can also go directly to the comparison area here. The page allows you to filter rate offers based on size of loan, your credit rating, your gross income, state, county and other options.

The service is currently only offered in the US and even there, not for every state. For example, you won’t find offers for Alaska (and consequently, those in Alaska won’t see comparison ads triggered on mortgage-related searches).

Fox told me there are a variety of reasons preventing universal coverage at the moment, such as some regulatory issues plus that they simply don’t have enough advertisers on board in some places to provide a robust product. Google’s also experimented with a similar product in the UK in 2008. That no longer runs, but what Google learned from the test there was applied to today’s US rollout, the company told me.

Ads are sold on a cost-per-lead basis. When someone clicks to receive a quote, the advertiser is forwarded the information and billed. The advertiser also receives no personal information about the person. In fact, they don’t even get the person’s real phone number. Google provides a temporary bridging number that connects the advertiser to the customer. After that, it’s up to the customer to provide their own “real” number if they want follow-up, Fox said.

Having gone through a remortgage recently, I raised the issue that often rates advertised don’t actually match what you get in the end. Nick Fox said Google’s done work to help eliminate this.

“These are not teaser rates. These are real rates based on what’s used in the industry,” he said, saying that the system pulls back from current data.

The system also uses IP detection to automatically guess at your area. When I tried it, it correctly preset the filters to my state and county.

While mortgages are being offered to start, Google expects other products eventually will be added.

“We don’t have anything to announce in terms of the future areas this will expand in. We’ll learn from the mortgages experiment, how well it is working, where it is working and based on that, we’ll have a better sense of other places to expand the product to,” Fox said.

The expected Google Music Search is real, and our Google Music Search 2.0 Launches With Musical “OneBox” story provides the play-by-play of how it works. Be sure to read that! From me, the color commentary out of today’s launch event at Capitol Records in Hollywood. I’ll be live blogging the news.

We’re expecting some musical stars, including guests from OneRepublic, Dead By Sunrise, Linkin Park and Mos Def. I’ll do my best to keep up, given my lack of popular musical culture makes me the butt of jokes to those who know me well. Have mercy, OK?

And it’s 4:04pm, with loud music playing, and a stranger intermix of tech folks and music folks. And nothing happening. So just stay patient. It’s “The Joker” playing, by the way. You know, midnight toker?

OK, a little video and Syd Schwartz, senior vice president of EMI takes the stage. Talking abou the history here in the Capitol tower some of the big artists. Me, I’m just afraid we’ll all die. Because it’s the only landmark building in LA and it always gets blown up in disaster movies.

Now Marissa Mayer from Google. “It’s clear to us that for our users, music holds a special and deal place.” Or a quote very similar to that.

Google mission to organize info. But quickly realized they needed more than web pages to do this. So image search in 2001. Book search in 2003. In 2005, Google Maps to search the physical world. 2007, Universal Search blended this all together.

How’s music fit in. Top 10 searches involve lyrics and musics. Hey, there’s a Hugh Grant and Drew Barrymore movie called Music & Lyrics. It’s awesome.

Marissa’s routine is to listen in the car, memorize a snippet of a song, then going back and searching for it. But that’s not easy. So, “why can’t Google make music more intuitive and easy to find online?” So excited about the new service today (HEY, READ OUR ARTICLE UP THERE IN THE FIRST PARAGRAPH FOR MORE ON THAT). Only service that lets you play an entire song. And Gracenote allows them to do full lyric search [don't ask about all those sites listing lyrics illegally in some cases in Google's regular results].

Now RJ Pittman to demo. Music in his blood, mother is a music teacher. One of his favorite bands is OneRepublic. But is it one word? Ryan from the band he jokes tells him that’s right. But excited you can do two words and find it that way. Now showing how you can get music one click away. Hey, OneRepublic is pretty good. I should download them. From Amazon, cause I like how they’re easy and DRM free. Wait, I mean from one of the Google partners that are involved with it. Don’t worry, see our article, Google doesn’t earn off that itself.

Searches for a lyric from a Police song. No, not that one. You were thinking Roxanne. Not that one. Long lyric, finds a OneBox, plays music. People applaud.

“If you ever wondered what a search result sounded like on Google, this is it,” he says, as he plays Dead By Sunrise.

Marissa’s back up and talking about MySpace and LaLa for live music streaming, music vendors imeem, Pandora & Rhapsody. And Gracenote for Lyrics.

Now Ali Partove SVP of BizDev for MySpace and Courtney Holt president of MySpace Music. Talking how they’ll be able to better get news out about tours. How the social graph can be useful to the music experience. Music has helped MySpace grow and this will help even more.

“Music on Google Search. How F-ing cool is that,” says Holt. Except he didn’t have that dash in there :) Hey, we’re at a rockish event gang.

Now Holt showing popup you get when you click on Google music results, allows you to by the MP3, watch the official video or get concert info. If you clickthru, you get to a dedicated music page at MySpace for the artist. Now he’s giving props to the Google team for building. “In my 7 years in digital music, coming up for an arrangement…” where consumer, artist, rightsholders and many others win is hard, but says Google’s done it. “the main beneficiary is the consumer” getting a better music experience.

Now Bill Nguyen, chairman and founder of LaLa. Or lala, I guess. “We all begin music discovery at the same place, Google Search.” Well no, but OK. And now he’s showing three bands that haven’t gotten discovered more. First is Edward Sharpe & The Magnetic Sharpe. They sound nice. I’ll download them.

Shows how on lala, you can go to their page and see who has discovered them as a fan, see other music that person likes, you can follow folks like Twitter. It’s Twitter for music lovers.

All built on the web because he hates MP3, doesn’t like to organize them. Me, I love them. I want to own my music hardcopy, I guess. But he shows how you can buy the music on their service, what you already own. I assume you can download. I assume you can then put on your MP3 player. Right?

Showing how you can preview a song and buy it if you like it. I kind of do this on Amazon already. But I think you can listen to the entire thing. So that’s cool. Now showing what looks like an iTunes to organize your music. But not probably as sucky as iTunes. And from lala, on the web.

RJ’s back. Goal was to keep the product simple. Marissa taught him sometimes greatest challenge is restraint in features. Make it so if you know a few words, boom, you can get that music in a click. [Hey, what about being able to sing it. Kind of like Shazaam. But there are other services out there like that, too. Let's have that, too!]

When you’re over 25, you stop listening to new music he’s heard. Yeah. [But then when you're past 40, you start feeling like you'd better listen again or you'll be really out of it. Just saying.] But he’s excited this will let people discover more. [Except, I've gotta say, there's not a lot of discovery here. It's all search. It works because you KNOW an artist or KNOW a lyric you heard. It's not suggesting new stuff you don't know].

Now a roundtable. We have Mos Def, joining Wendy Nussbaum of Universal Music Group, Steve Savoca of Domino Recording, Mike Shinoda of Linkin Park, Ryan Tedder of OneRepublic and Syd Schwartz from EMI.

Asks Mike about how things have changed over time. Mike talks about how Linkin Park had a different name, found the name Lincoln Park but couldn’t get that domain. “We thought, this could be an important thing to have a dot.com of our name.” So we literally changed the name of the band to get the URL that would give us the most direct link to our fans. [and yes, I know he also said this yesterday at the 140 Twitter conference. He mentioned to everyone here he told the story there, too].

RJ asks if they thought about getting non-US domain names for the band, like in France. Mike says thought about that.

Wendy now talking about how this will lead people to legitimate sources of music [and I hear Apple thinking now ugh, maybe you know holding all our legit music behind that absurd you've gotta have iTunes to find it AppleWall is something we ought to take down].

RJ asks about the idea of seeing what people share, he’d love to see what Mos is listening to, as Linkin Park and OneRepublic. Wendy says basically sure, always thinking about new ways.

Syd now talking about how he wanted to build out his jazz collection, going to Tower Records and seeing Donald Fagan of Steely Dan. Followed him around as he’s a jazz expert. “So trying to not look stalkerish,” he followed him for 1 1/2 hours. “I discoverd some great stuff .. but I look at what’s been presented here today and think wow, I could have saved myself a restraining order.”

Mos asked about what gets him excited in terms of the web and technology with what he does.

Says sure, for all artists, the internet and Google and YouTube and things like that have been incredible important. And now a billion cameras going off with high speed shutters. Thought we were all digital these days.

“Me, to be perfectly honest, I’m a child of hte 70s, so I walk around every day thinking I’m in Buck Rodgers or Battlestar Galactica … I’m still really getting over the cell phone.”

See now similar to early 20th century with new tech, a wide open field. … “I’m still getting over email. Wow, it’s incredible.” Excited at opp for independents and how quickly they can turn music around.

Steve says excited to have independent have a seat at the table. “This means a lot to us. What we do is niche.” Digital is 50% of their business and primarily get people through word of mouth. “This is a zero friction music experience.” Hear about an artist, hear the artist. With things like lala and OneBox, feel new opportunities to get their music out

“We’ve got to change consumption behavior.” Need environment where people can access these things. When we can expose people to these great services we can change behavior and that the legitimate side of things is where to be.

Ryan talking about how he feels MySpace is what made their breakthrough. He got dropped same day as Katy Perry and Jonas Brothers from the same label. Audience goes wow, because even I know they’re all hot. Thought if he had to make poster and put them up, “I’m screwed.” Found MySpace thought it was perfect, free. Would search for everyone 18-22 on MySpace and emailed them when in town to get them out to concerts. “Were it not for MySpace … that would not happen if it weren’t for technology as it is [people coming out].” For Google, when type in a song until today are bittorrent illegal download sites. [which is kind of bad for Google, because if that's true, well, they're still listing all that stuff]. He’s excited that it literally focuses stuff.

OneRepublic has new album coming. Any new plans in the digital space? Yes, might do something with Twitter. “I don’t know what artists did 20 years ago .. did you rent out the Rainbow Room and throw a kegger.” I’m loving Ryan now, because he’s so totally using this entire new medium. “I kind of have two jobs, writing and producing .. I need things that won’t take up the whole day.” “It’s a blessing and a curse, because we have to be in 20 countries at once.”

Mike from Linkin Park says you’ll be able to see how they make their next album online.

And now we’re going to do Q&A.

How will this change industry in future? Courtney says Google has lots of queries but hasn’t been leading people to a legit experience. This gives consumer more uniformity. It actually gives the rightsholder more control, a way to make money.

Mike from Linkin Park, “I want as few steps as possible between me and that think I’m looking for, the band … I want to be able to find them amidst the noise.” This is what excites him most about this.

Bill talking about how this will let people hear more diverse music. “They’re going to do for music what they did for the web.”

What about outside the US and streaming. RJ says right, only US right now. Focus is to go heads down and tackle this market.

I asked Ryan and Courtney about how they’ve both mentioned there being illegit content on Google, and how that’s not going away, so what do the think Google should do. Or is being at the top of the list enough?

Ryan:

“I’m not entirely sure that it’s Google’s problem … I think it’s a huge step foward … first major company outside of a record label that filters through the noise … again that was the first thing that caught my attention. This is the first internet anything that does that … I don’t know how they did it. You’ve got people way smarter than us [says looking at Marissa and RJ of Google] … You didn’t have to do it. .. as for the fans … clicking on dead links [is bad] … now you’re playing the song two seconds after typing in a lyric.”

Courtney:

“The number one way for the music industry to battle piracy is to make the legitimate image options better … here what Google’s done is made … an extremely convenient experience … also the speed of it … google has had an enormous focus on making it really freakin fast … I honestly believe simply becasue it is more ocnvenient and it’s from a brand that’s more [recognized, I think, he said]” will get it used.

Question on costs. If this generates more listening, more costs? Mike for lala says “more listening is more buying.”

“They don’t want to talk about it … they’re not going to tell you” Mos says, as another question comes up about buying and selling and how the music reps haven’t wanted to get into that.

Mos also impressed that labels seem to have jumped because this is Google [idea that Google is being, versus Napster .. I'll try to find another live blog account that caught this part better]. Live blogging is hard, did I say that?

And that’s it, the announcements all done. Techmeme has massive related coverage, both live plus regular articles that will be coming. Check it out. And rock and roll, people! Heh.

It’s Yahoo Investor Day, where Yahoo is briefing institutional investors and financial analysts on the company. I’m here for the morning portion, and I’ll be live blogging out of it. You can also watch the live web cast here.

Bartz Welcomes

Carol Bartz, CEO, on stage. Saying thing they have great opportunity. Lots to fix. Low return. “Today is the journey back to respect. We’re not here to wow you, but we’re here to intrigue you and earn your respect.”

What happened? Yahoo was the big shining star, then wasn’t. Wants to talk of “slice” of Yahoo that’s not search or display or clicks or CPM that but “the scale of what Yahoo is. The importance … the real intensity of what Yahoo is.”

To define what Yahoo is, they went out to talk to users around the world in 10 countries. “And they were kind of confused and said what’s Yahoo.” Said it was where they go to check in … it’s the combination of my world and the world [OK, we know they didn't say it that way." Says sure, they go to Facebook and elsewhere. "But guess what? They come home. They come home to Yahoo ... that's scale no one else has. That's diversity no one else has."

Something about advertisers. Didn't catch it because apparently you can't video what's being live webcast. As someone who came over to tell me. So couldn't hear her.

"We can run a more complex business because we're so huge." "We are the largest communications engine in the world." [first time I've heard "communications engine" phrase and like it. Finally, she's defining Yahoo in a way that would fit in a Twitter box.

So Yahoo has scale. And fallen flat on its face at times, "Sure." But they learn by that.

"Carol who's your competition? Who should we compare you to." Sport analysts look at ESPN; entertainment folks look at TMZ. Apparently no one compares them to Google or she's not saying. "We make up every morning with passion to beat every one of them ... and we usually do .. we're not a search company. we're not a display company. we're a broadbased internet [something, communication?] company … that is awesome.”

Yahoo also had diversity. And now she’s lining up three “labs” speaker to come. They’re amazing.

Yahoo has great editorial, human brains, combine great human editorial with great machine learning. So that makes us unique.

Sales force also makes them unique. Yahoo a “high touch” company.

The last thing that makes us unique is that we have fallen, and we really want to get back up. Up on our tippy toes … if you haven’t had good times and bad times, you don’t know what you’re doing. We’ve had good times and bad times. We prefer the good times [chuckles in room]” Going to get back to that.

Not here to make big promises. We’re here to impress you … about how we innovate, our scale.

Thanks everyone and brings up Ari Balogh, Yahoo CTO.

Balogh

Brands matter. Large brands are being found to be more important between 2003 to 2009. There’s a pie chart to prove this. Yahoo’s a big brand, so they have opportunity.

Another chart showing how time online has grown. I don’t dare take a picture of it, though. Email up 18% since 2003, search up 12%, video up 63%.

A bar chart of mobile internet users since 2008. 490 million worldwide to 596 now to 1,453 million projected in 2010 from comScore and eMarketer. No mention if they’ll all be using Android, but the Kindle gets a call out.

Other big trend is how traditional silos are changing. Talks about Yahoo Buzz now. Place to find stuff. You can search. Is Twitter real time search [with tone that no, it's not]. YouTube, video or search? The point is that it’s really about personal social utility. I think we’re being told don’t worry about all those real time search deals. See, we kind of have it in Yahoo Buzz.

Now Yahoo’s [OK, Yahoo!'s] Value Proposition. There’s a flow chart, but we’re on to a page on Buzz about the Chupacabra monster being caught? That was an awesome X Files. C’mon you remember it. Kids, the X Files was a TV show with Hank Moody from Californication and Scully, who like lives in England now. But I digress…

A Yahoo editor ssaw this story and decided to put it on the home page because it was unique enough. And it takes a small village to make that sort of thing happen [which seems a village too much]. “The results were impressive.” 15 million people read it, 30,000 votes, 2,000 comments, 700,000 searches. “This drove engagement downstream in Yahoo.” [and did the site pay for the placement, I wonder? Because if so, it drove some direct revenue. But it might have been a pure editorial pick].

So Yahoo does this day after day, content across multiple platforms. No other company can combine that great technology with the right voice. Yahoo creates “well lit contexts” for advertisers to connect with their audience.

Yahoo learns more and more about how people behave and how they can sell different segments to advertisers.

Yahoo execution strategy starts with deep personal relevance. Generates “insights at scale” that improve experience with people and advertiser.

Now Tapan Bhat, Senior Vice President, Integrated Consumer Experiences, comes to talk about the new Yahoo  home page.

Yahoo’s Home Page

345 million visitors worldwide and other big numbers I didn’t catch. Wanted new home page to give best of web to grow audience, giving search. The whole my world and the world thing.

Talking about how as entered counties, they build different ones and ended up with 44 different pages based on 30 different coding [or close to that]. Point is, lots of cost, especially as something always tested on the home page, so has to be implemented after a reaction in market A, then add to market B could take a year to fully roll out and improve the user experience everywhere. “And that’s dumb.”

Talking “page yield” concept, what’s value of the traffic a page sends, how much people come back, testing how to improve page yield. Again, takes another year to roll something that works in one market out further. New platform means an immediate worldwide rollout. Great value there.

End 2010, 33 different code bases will be down to 1.

My Favorites section of home page now My Applications, as part of first version. And other changes, and people we’re overwhelmed. Not emotional reaction Yahoo wanted. But turns out, the color wasn’t what people expected. So lighted the blue. “You’d be surprised. Immediately we saw a change in the metrics.”

Created the My Mailboxes area that can bring everything into one place. People weren’t using it, though. People wanted the mail boxes accessible through Yahoo but separate (Yahoo Mail separate from AOL Mail separate from Gmail, not them all flowing into one box).

Talking about common inbox, elevating messages from those most important in your lives (which Gina Trapani wants, so either Yahoo hasn’t convinced her or need to do more outreach!).

“Is the home page done? No, the home page is never done .. when we get to a threshold, it’s what’s next?”

Showing a hover preview as you scrolled down your apps, say you hover over your Flickr link on the home page, see a preview. People complained about “pop-ups.” Turns out people read by navigating with their mice. They move the mouse over to text to help them read. So if you put an overlay that shows up, “disconcerting for them.” So changed the delay between when you hover over a link and get an overlay from 150 milliseconds to 450 millisecond and had a big impact.

Search! “The biggest driver of page yield is search.” Search queries monetize at a high rate. Navigation and search is a core need. So tweaked with the search box. Made the button said “web search.” Made the button bigger and the search box bigger. “We sat people down, and they said, ‘Where’s search’?” [But uh oh, they'll find it much easier over at Google and Bing].

Turns out highlight web search box also highlighted other things [didn't catch them, maybe the other search verticles like Images and Video].

Looked at page width on how drove search. We came to this realization that less is more. When you have less on the page, the things that are left matter more. So home page, went to a “short” home page. But people expected search to be a “fundamental” part of what they do online. So redesigned search page.

Trying to have a more relevant experience there. Talking about Yahoo’s new filters (see also here) and how you can search and then get to places like Hulu (and now you can get to Google!)

How’s home page going? 9% increase in page views, 20% increase in time spent. What’s causing this? Ability to add applications from any site on the web and featured content. [Yahoo's got an app for that. Well, a place for you to park your app. Oh, I've got it. The new slogan. Yahoo: Your iPhone For the Web].

Yahoo’s got a content optimization engine that gets it to the “right user and the right time” [man, I hate any catchprase that talks about right blah at the right whatever. Like doing the right thing is a given, not a selling point. Doing more than right, that's a selling point].

Speaking of selling, came up with way to sell ads in the overlays you get when you click on something in your favorites. Also sells applications as something people can add to their favorites.

Bryan Lamkin now comes up.

Bryan manages the applications group. Apps drive engagement. Mail up compared to MSN or Gmail. 100 billion messages received in Yahoo Mail per month. 81+ billion instant messages sent each month. 4+ billion photos uploaded to Flickr. Yahoo Answers has 30 million question and answers posted each month [we'll sometimes the answers don't really answer, but still, Yahoo Answers has lots of goodness there]. Yahoo Groups has 120 million members.

Lots of handset / mobile growth. So bringing mail experience to more phones. Mail and messaging are number one apps people want. Flickr app now one of the most popular social apps on iPhone [and took Yahoo long enough to roll it out]. Many who have messenger on their phones never used it on the desktop [looking for ways to save on text messaging costs in a down economy, eh?].

Talking of cutting spam in inboxes. Need people have for someone to do trusted filtering. “We deliver more photos via mail in a week than get posted on Facebook in a month”

Talking about smart inbox that elevates messages from those you care about. [Precisely what tech blogger Gina Trapani wants. So Yahoo hasn't convinced her or needs to do more outreach!].

Lots on how you can do things like view your photos in mail, interact with Yahoo Groups, watch your Netflix download in mail. [Mail, your desktop for the world!].

Yahoo Mail represents 84% of Yahoo’s logged in users. And logged in users in general far more engaged on Yahoo.

James Pitaro now up.

Media Business

James telling story about how he didn’t wear a tie to meet the NBA commissioner with Carol. She told him, “where’s your tie.” He’s like Yahoo doesn’t roll that way. Met the commissioner who said, “where’s your tie?” He thought the was being Punked. Point? See Yahoo’s learning [he's wearing a tie on stage].

What’s Yahoo Media? News, Sports, Finances, Games — these properties are those identified as core to users and so core to Yahoo.

Strategy is to grow the products and engagement by delivering premium content by innovative and insightful buzzword buzzword. And something about relevancy.

Sept, had 80 million unique users, #1 in lots of verticals areas. “When Life Happens, People Come To Yahoo.” text on slide behind him.

News hits, people come to Yahoo. Both breaking news and Yahoo Finance up as, well, people’s finances are down. The down economy ironically has generated biggest audience ever for a financial site. Michael Jackson’s death, well, let’s say Yahoo did well in page views by it.

Search used to figure out what content is needed.

Talks about the guy who caught a baseball and gave it to his daughter who threw it back by mistake [it was really sweet]. Build a blog around it. More traffic.

Yahoo also has original news content. So other news organization take them seriously [though oddly don't seem to mind that Yahoo then competes with them, I guess].

They produce their own video. [and i'm having serious connectivity issues, so i won't be keeping up for a bit, sorry] Yahoo’s not going to make sitcoms [phew]

Publishers should look at Yahoo not as a competitor but a partner. Can drive traffic to them. Rising tide, boats go up.

Want exclusives. Get exclusives movie clips for free, which he says is “amazing” but they get lots of traffic. Same with sports clips from leagues.

They”ll never have it all, so they have to be open and link out. Artist pages, you can create your own experience and linked to whatever want. Since launched, engagement and view up. Despite being open. [um, maybe you're also promoting your pages for directly? and being open by linking out by making the users themselves have to do it? let's not go overboard on how giving Yahoo is there. And side point, lots of talk about licensing. Those licenses, which is something Google doesn't do because it doesn't use content so extensively, are still what help keep lots of media companies loving Yahoo and hating Google].

Dunkin Donuts partnered to do a mornign sports minutes. Fantasy Football Nation helped bring sports scores for free to Yahoo’s mobile app.

And it’s break time. When we come back …. Search!

Rinse & Repeat / Listen & Learn

Break is over. Had trouble because Carol Bartz was standing in front of my chair in the audience. So I said Carol, get the hell out of my way. And she told me to f-off with a wink. No, no, silly fools. I waited until she left. But that would have been funny. Or maybe I’m a bit tired. You know, the day started pretty early.

Ari is back and talking about scale, iteration, getting feedback. And I’m feeling all buzzwordy, despite liking Ari from earlier (he’s new to Yahoo). How about rinse & repeat? Feedback & Iteration? No, Rinse & Repeat. Or Listen & Learn. OK, maybe I shouldn’t be in charge of catchphrases.

Anyway, Yahoo does lots of “bucket tests” that can involve millions of people. “That generates an amazing amount of data for us. And we learn.”

What’s a petabyte of data [one quadrillon bytes]? What’s that feel like [I don't know, but it sounds terrible, like a pedophile]. Anyway, they’ve got lots of data. And big believers in the cloud. In fact, they were into the cloud before it became a current fad topic [IE Google, we were in the cloud before you, so stop yapping like you're the cloud makers. See, I say these unsaid things. Sometimes I even say them correctly].

Running down the home page, talking about load balancing of apps. Content optimization, studying what works on home page and what to promote. Churning search data to know what popular searches to highlight. Ad optimization [can they optimize that annoying Progressive Direct woman and her ad off the home page?]

“Cloud computing is not some big sexy thing we’re trying to sell out there … it’s how we do business.”

“We bring science, and we move science forward” to the changes they make. “Computational advertising” is bringing science of trends to it.

Search

Now we’ve got Prabhakar Raghavan who heads Yahoo Research.

Search is a virtuous cycle. How users use search to get across the Yahoo network and what do they do to get back to Yahoo search. And how does the underlying technology play.

“We’ve gotten awfully good to develop search technology just for search.” But how can it be used to enhance user experience elsewhere, such as on the home page.

Talking about the 10 blue links experience [kaching! more money for former Ask CEO Jim Lanzone who never gets credited for popularizing this term].

Users want more than that. He contends people don’t want to search. “People don’t get home and say let me run some searches.” They want to figure out where to eat dinner, to find an apartment, accomplish other tasks. But search paradigm forces people away from act of doing things to having to do research.

Looking now at Brett Favre on Yahoo, showing a smart box with incomprehensive football stats to me [because I'm a football moron -- if you love football, you probably love this]. In fact, it’s showing the exact stuff like next game, injury status and other stuff people typically search for. “It’s the kind of the thing in the past that the user would have to go to many places to get … but here it is in one place, on Yahoo.”And good choice of query. Bing and Google have many smart boxes with direct answers on some topics, but nothing for this particular query.

Web of pages versus web of things, he says, and has illustration showing how Yahoo extracted all this information from pages (some licensed; the box doesn’t say source. In fact, the box really draws from Yahoo Sports rather than from the open web of content, as best I can tell).

We have to move away from a web of pages. Crawling the web, that’s like a commodity. Have to lift above that. Which is kind of true but also on target for the overall message that Yahoo doesn’t really need its own expensive search tech but instead can lease from Microsoft.

Looking at current query smart box for Madonna. Now looking at a future [one that still has Madonna, I guess]

It looks nice. The entire page has things like Yahoo News results, video stuff at the top. “no blue links” Bank of America page. Shows latest news at the top, quote results in the middle, blog matches, local results. [and this is all good. but it's also risky, because you might guess wrong. that's main reason folk haven't made this type of deep dive so far. "this isn't your father's search experience. This is the type of thing we have to get really good to deliver in the future.

Content comes from the open web but also licensed [not said, see, we have premium content Google won't have because they don't license. And they can also tell good content within Yahoo's own network based on activity [but worrisome, because you really don't want them doing what Lycos used to do, where you could never escape Lycos when doing a search.]

Showing ideas on integrating instant messaging. Showing how you can maybe email about going to a movie and having results show up next to your email where you can copy movie times over. Which would be handy, I suppose — but I’m pretty sure Bing’s been doing that for about six months now. So it’s not a huge driver.

Wish I had screenshots of some of the mockups. But then, they’re also mockups for now. Let’s get them out there, see how they do.

Testing Testing Testing

Now Yahoo economist Preston McAfree up talking about algorithms and content optimization. Figuring out based on demographic of visitors what they want to see on the Yahoo home page.

This also helped editors get better in learning what stories feed what groups. That they don’t have stories that target particular users. So virtuous cycle — jokes every speaker today required to use that phrase — but does. Says growing recognition that as web gets bigger, editors play an important role. Algorithms are easily gained. If done purely like that, people will design to fool that. And human learning.

Digg, the quitessential place algo is used, he says, has started to use human editors. Surprised he didn’t say Google because that’s sure who this felt aimed at

Talking how putting a second ad below the first search ad at top of results increases clicks on first ad, doesn’t decrease it. Big surprise to them. This type of rigorous experimentation have helped them learn. Works until you have more than four ads, then clicks go down. So next time you complain about all those ads at the top of Yahoo? Hey, they’re paying the bills. Actually, doesn’t talk about what all those ads at the top of the page do in terms of overall relevancy, clicks on the natural listings.

Missing some now, sorry, because the wireless is going flakey again. And my usually dependable Verizon card isn’t always getting a good signal.

Joy of Yahoo is that they get huge samples to get statistically significant stuff.

Virtual works and “real world” becoming one and the same. Virtual world can influence your purchasing in the real world weeks later.

50 years after printing press was invented, most of the influence yet to be felt. Novels, billboards, newspapers — way forward in the future. As was much technology. “I believe the internet will have at least as large an effect on society … as the printing press .. and like the printing press … almost all those inventions and discoveries lie in the future.”

Innovation

Ari’s back, stressing that we should see how much talent and vision that Yahoo has now. And now he’s talking about how, in fact, that Yahoo is an “innovation engine.”

Again references of investing in cloud not because its sexy but helps with business.

Goal is to get all products to a cycle time of weeks [IE, hey investors, look at how quickly we'll be able to churn knowledge of what users want into delivering that content, along with ads. We're gonna be unmatched! We're alread unmatched! Which, by the way, for a content company may well be true].

And Carol’s on stage. Congrats her product people for presentations. “If anyone goes away from this and says is Yahoo a technology company? Then we’ve missed.”

Says nothing shown is promises with exception of those search mocks. One more presentation before BlackBerry break “Although you don’t need it, because you’re already doing it.” Laughs all around.

Keith Nilssoon coming up exec VP for developing countries.

International

Only 11.5% of ads are international today (IE, outside the North America). Didn’t catch if that’s online ads overall or just for Yahoo.

75% of Yahoo users come from international (IE, outside the North America. I mean there’s a slide that shows a pie that’s part international and part North America, as if North America isn’t part of the international community. But I know, I know, I get what they mean).

They’ve had a decentralized team with poor technology sharing so you’d get all those source code version and product inconsistencies “that effected our brand, our engagement with users and ultimately our ability to monetize.”

With work on local, became top cricket site in India. In Singapore (I think) surpass news sites like CNN & BBC.

Talk about doing some ads deals. Not covering what a mess it is that to buy paid search in various countries, you have to open separate accounts for each of them (at Google, one account can target the world). But that’ll get fixed with the Bing-Yahoo deal, so let’s not dwell on it.

Emerging markets only 2.5% of ad spend is online right now. “The battle for these emerging markets is happening today.” Companies that get in will be able to drive growth for some time.

So Back To Basic strategy. Give stuff on PCs and phones. Improve communication tools, email and IM. Global home page with local content. Know the markets are there. They’ll come online.

Keys to win? Mobile. 4X times more mobile users than internet users in emerging markets. In India, it’s 8X. So focusing on lightweight mobile friendly content. Partnering with internet cafes.

And we’re done. And, I’ve got to sadly depart the event. It continues through the afternoon with a focus on ad strategy and marketplaces and a financial overview. You can watch live later today here.

Overall, good job, Yahoo. The presentations do stress what a leading company you are (as I’ve said you are, see Yahoo The Failure: Myth Versus Reality). Let’s see if the analysts will move past the focus on “so what about search” that still seems to grip them at these types of presentations.

See also related discussion and articles on Techmeme.

Google Social Search is rolling out, a new service from Google that allows you to easily find material written by people you know and trust. It’s a pretty cool idea, especially in that it’s pretty painless to get started using it. The service will be available through Google Labs Experimental later today. Below, a look at the service.

It’s Not Twitter Search; It’s Not Real Time Search

Let’s first talk about what Social Search is not, to avoid confusion, especially after all the seemingly related news that’s come out recently.

Last Wednesday, Bing announced a deal with Twitter to produce Bing Twitter Search (see Up Close With Bing’s Twitter Search Engine). The new service allows you to search for matching tweets or find interesting links that have been tweeted.

That same day, Google announced that it also had a deal with Twitter to tap into Twitter’s information and make use of it in some way. Potentially, it might produce a Twitter search engine of its own or make a broader real time search engine.

Also that day, Google announced that “Google Social Search” would be coming. This was in development before the Twitter deal was announced and, in fact, doesn’t depend on it at all. Social search doesn’t search against tweets, as I’ll explain.

Our Google Social Search Is Coming & More On Google-Twitter article provides further background on those two separate announcements that Google made last week. What Is Real Time Search? Definitions & Players also provides background on the area of real time search, for those that are interested. But the important thing is….

GOOGLE SOCIAL SEARCH IS NOT REAL TIME SEARCH

That’s not bad, by the way. In fact, Google Social Search is pretty cool. It’s just important that it doesn’t get confused with a completely different type of search engine.

Social Search Is Trusted Search

So what is Google Social Search? It’s a way that Google figures out people you trust, then ensures that you see content from them showing up in your search results.

That sounds like a pretty simple idea, and companies have approached this in various ways (see Search 4.0: Social Search Engines & Putting Humans Back In Search). Typically, the social concepts for refining results have been to allow people to form social networks and then:

• Refine search results based on actual search activity in that network
• Share results among each other
• Define only certain sites that should be included in results

The first two have privacy concerns, among other issues. The last two especially involved work. You’ve got to actively chose to share results or actively define a set of sites you want to search against. Who wants to do all this?

With Google Social Search, there’s still some work required. But it’s minimal if you already make use of Twitter, Flickr or an existing public social network. Heck, if you use Gmail or Google Reader, you may already be social search ready.

Social Search In Action

Let’s look at how social search actually works in terms of listings, then we’ll spin back to examine what happens under the hood.

Here’s a search I did for Newport Beach:

These results appeared at the bottom of the search page, in their own section, with the heading “Results from people in your social circle for newport beach” as the first arrow highlights. Underneath that are some results from people I’m connected with socially.

The first is myself, my own blog is listed. That makes sense, in a way. I know myself, and I write about Newport Beach a lot.

The second result is from Thomas Hawk. The second arrow highlights why Google is showing this. It sees that Thomas and I are connected through Flickr, that we’re mutual contacts there.

The second results also highlights something Google touts about social search. You won’t always get these results showing up, but when you do, Google feels they’ll be pretty interesting to you.

In this case, it was. I had no idea that Thomas had such a nice collection of Newport Beach photos.

Drilling Into Social Search

You’re not limited to what shows in your main results. For example, in a search on JetBlue, I get this:

If I click on that top “Results from” link, that the arrow points at, I get to see a full list of social search results like this:

Notice how the first arrow points to “Social” in the Google Search Options area. This tells you that “social” results are showing, versus video results, blog results, news results or other times that search options can show (see Up Close With Google Search Options for more about the entire search options area and how it works).

The second arrow shows that the social results come from “all people” of those within your social circle deemed to have relevant content on the topic. You can click on a name and drill down further, if you like. For example:

That drilldown shows me content from Eric Goldman related to “jetblue,” and the arrow tells me I’m connected to him through Gmail.

This drilldown shows content from David Wallace and that I’m connected to him through Digg.

In the first two examples, both people had one post they’d written that that was relevant to JetBlue. But in this case:

You see that 10e20 has two posts (and that I’m connected with them via Twitter).

Note that sometimes results from people will show, but you won’t see their name mentioned. For example:

See the boxed result? There’s no corresponding person for that listing. This is because it’s a match drawn out of my Google Reader account (see how it says “Reader Subscription) in the listing?).

Also notice that people are listed on the left roughly in the same order as their actual posts are shown. That’s supposed to be how it works, but there are bugs that sometimes mess this up. That’s why “David Berkowitz” is listed second in the people column but his actual post in the results doesn’t appear until the bottom of the page. Google told me they’re working on this.

Finally, the people you see listed for one query will change if you do another. That’s because the people shown depends on which pages Google decides are relevant for a specific query, which raises the issue of how things are ranked.

Ranking, Sorting & Directly Using Social Search

So what comes first in social search? Google says its using its normal ranking algorithm to pick content, so factors such as the number of people linking to a page, the quality of links to that page, the content of the page itself and so on all can have a factor.

Down the line, presumably Google could do more, such as look at the links only between pages authored by those in your social network or by trying to assign an authority score to those you know, based on how closely you seem to be associated with them. But that’s not happening now.

By the way, want to use Google Social Search without hoping it just happens to show up at the bottom of your regular results? After you do any search, use the “Show Options” box just under the search box at the top of the search results page. Then select the “Social” link along the left-side of the page. Now you’ve got social search results.

Your Google Social Circle

One thing I love about the new service is how it makes use of the “social circle” term rather than “social graph,” a phrase more popular in 2007 and 2008 but which doesn’t really explain much to people. Social circle makes sense — these are people you are connected with. They’re in your “circle” of friends.

But how does Google know what your social circle is, in order to produce the social search results? Three methods, the company told me, when I talked with Google about the service:

• Your Google Reader account
• Your Google Chat / Gmail Contacts
• Your Google Profile

Google Reader is pretty easy to understand. If you subscribe to blogs using Google Reader, Google figures you like or trust content from those blogs, so they effectively become part of your social circle.

Google Chat is harder to get your head around. If you have a Gmail account, you have a “Chat” section on the left-hand side of your screen. Anyone you’ve enabled chat for is considered a trusted contact by Google, so they become part of your social circle.

But wait, there’s more! Gmail also has a contacts area on the left-hand side of the page, where you can classify people as friends, family or coworkers. Putting someone into one of these categories makes them part of your social circle, as well.

But wait, there’s … confusion. Above, I showed an example of how Eric Goldman is part of my social circle through Gmail. The problem is, he’s not one of my chat contacts. I’ve not tagged him in contacts into the friends, family or coworkers categories. He IS a Gmail contact, so in his particular case, that seems to be enough to put him in my social circle.

Also keep in mind that if you use a Google Apps account (as I do) for email, chances are, social search won’t tap into your email contacts. That’s because to use social search, you need to be logged in using a Google Account. That Google Account will have a contact list that is separate from your Google Apps contacts.

Finally, Google Profiles are used to form your circle. Our Hoping To Improve People Search, Google Launches “Profile Results” article explains how Google Profiles work in more depth. The short story is that if you list a social network you belong to on your profile, Google can use that to effectively import those people into your social circle.

For example, on my Google Profile, I list things like my Twitter account, my Flickr account and my Digg account. Google can then look at these accounts and know who my contacts there are. That’s why in the examples above, you can see Google reporting that I was connected to someone through Digg or Twitter and so on.

Your Social Circle & Knowing Their Content

Knowing your social circle is one thing. Knowing what they’ve written is another. The magic behind the scenes all comes down to links. Here’s an example scenario:

1. Google sees I have a friend on Twitter
2. That friend links to their blog from their Twitter profile
3. Google understands that they are connected to that blog
4. The friend’s blog has a link to their Flickr account
5. Google may understand, then, that the person I know on Twitter is also related to their Flickr account, even if that account wasn’t listed on their Twitter profile

To understand more about how Google can form these connections, see our past article, Mine The Web’s Socially-Tagged Links: Google Social Graph API Launched.

Privacy Issues

One of the nice things with social search is that the privacy issues seem minor. No one but you will see your “friend” results, because those results are expressly tailored to your network.

More important, there’s no personal information from your friends being shared. Social search is simply based on figuring how who your friends are and what content they produce, all of which is already in public.

Things that aren’t in public don’t show in social search. Right now, Google has no way of discovering who your Facebook friends are or content on Facebook that you’ve authored, if you’ve not somehow made that public.

Of course, in order to use the service at all, you have to be logged in. Our Google Search History Expands, Becomes Web History article gets into some of the issues here, although you don’t have to use the web history service itself to use social search.

More Behind The Scenes

To understand more about how social search works, there’s a movie. Two. From Google. The first covers how social search works from a user’s perspective and the second, how things operate behind the scenes:

Wishes

The service is experimental right now, so expect bugs, if you use it. One of the biggest issues I have with it is that you can’t see everyone Google has decided is part of your social circle. I wanted a social circle page to show me everyone. What if I want to remove someone? Or add someone? That’s something Google told me they’re considering for the future.

I also found odd things, such as when I did a search on “SEO,” I didn’t get matches from Matt Cutts, even though I know he’s part of my social network and writes much on that topic. A search on FriendFeed didn’t bring up Robert Scoble, who I’m connected to and who has written extensively on the topic. A feature allowing you to pick a particular person and search just for content from them would be pretty nice.

Also, often images of people or logos don’t show alongside their listings. Google’s aware of this, I was told, and it’s another bug they’re working on.

Overall, though, I’m pretty excited about the new feature. It’s one of those few things that when I saw it back in September, I immediately thought, “Wow — that’s something I’ll use.”

In addition, in a time when people have been showing more interest in ways to get back answers from trusted people they know (see The Rise Of Help Engines: Twitter & Aardvark), Google’s offering an interesting new solution to the problem.

Also see Techmeme for related discussion and coverage.

Too much time is often spent about the new features the various major search engines roll out or the latest deals they cut. Here at Search Engine Land, we can be as guilty of that as anyone. To correct it, I’ll be spending more and more time highlighting poor quality search results that I encounter, in hopes of nudging the industry to improve things.

Canaries In The Search Mine

I’ve spoken and written for years that when it comes to search engines, I think there are two “canaries in the coal mine” that catch a whiff of something bad emanating from the search engines.

The first are librarians and research professionals. They’re acutely aware of when search counts don’t make sense, if something important in a field they know isn’t being listed and other issues.

The far larger group are site owners or search marketers. The common joke is that when they spot search engine spam, that stands for Someone Positioned Above Me. Thus, it’s easy to dismiss what they see as just being colored by self-interest.

Sure, there’s some of that. But these people are also often subject experts. As surely as Cypher in the Matrix could look at computer code and say, “All I see now is blonde, brunette, redhead,” a subject expert like a site owner or search marketer can look at results and know when they don’t smell right, when something’s wrong.

About two weeks ago, our Reviewing Some Bad Google Search Results With Sergey Brin article highlighted a few bad results I could see in my subject area of expertise, that of search engines. Today, I’ll bring in another example, that of “search term research.”

Benchmarking Against Expert Knowledge

Search term research is one of the core aspects to search marketing, and I’ve covered various tools out there for years. Here at Search Engine Land, we maintain a Search Term Research page devoted to the topic. It’s a good page. There are probably better ones, and maintaining these types of pages is always difficult. Still, it’s kind of a benchmark for me. If I don’t see it ranking, what’s the quality of stuff that IS ranking above it?

As it turns out, our page isn’t on Google at all. Not at all. And it’s, um, our fault. The All In One SEO Pack plug-in we use with WordPress here set all of our category pages to be excluded from Google. It wasn’t that way originally. Back in the summer, the latest version of the plug-in changed things to overwrite how you’d previously configured it. I should have known better, too, because I even retweeted a warning about this. Everything’s fixed today, and we’ll see how things go.

Still, that page as well as my own knowledge of the area gives me a good benchmark I can use against other pages that do appear in the search results. So how’s it look?

Google: How About Some China Wholesale?

Over at Google, a search for search term research leads off with the Google AdWords Keyword Tool, which is an excellent first choice. It’s a dependable tool, offered for free, with great data. Some more tools follow that, then two older articles (from 2007 & 2006, respectively) on conference presentations about the topic. Those are kind of iffy to be in the top results given their age, but certainly they’re relevant. Then I get another tool, a fresher article that’s not super-substantial, a compilation list of articles and a nice conference presentation.

Overall, it’s not bad. Not fantastic, but not bad. Where things really fall down is when you go to the second page of results.

OK, few people go past the first page of results. I know this. But still, that second page of results? It contains what Google is presenting as among the very best out of 126 million pages on the web for this topic. The very best. And we get on the second page?

For those who can’t see the image above for some reason, the rundown:

• Link to a keyword research tool, which makes sense
• Really weird local results about local companies in New York that somehow seem related to search term research
• A really bad directory listing of resources
• An OK page listing some tools
• Agenda for a session on the topic for a conference in 2006
• A press release from someone speaking on the topic in 2006
• Another keyword research tool
• The most amazing bad result, some “China Wholesale Supplier” with search term research products. More on this in a bit…
• A review of one particular tool from 2006
• An article I wrote on the topic in 2007

I think the amazingly poor quality of these results are self-evident. Let’s look at that China Wholesale page. Again, out of 126 million possible matches, this is what Google thinks is the 18th best out of all of them for the topic of search term research:

I can’t even figure out what this is! One arrow points to how the page is in the “Search Term Research” category of the hosting web site. The other two point at what’s listed in this category, oil paintings of some soccer stars.

How on earth has Google, with its supposedly awesome attention to search quality, allowed this to show so high in the results?

But Bing’s Worse!

At least Google can fall back on the “others are worse” excuse. Let’s go to Bing:

Ugh. The rundown:

• Terms of service for those looking into broadband research?
• A tool, OK
• Wikipedia article on research in general. Not search term research — just research
• An undated article with bad advice that the best way to do research isn’t to do it at all. Just write! See what terms generate visits after you write. That’s terrible advice, because if you haven’t written using important terms, you’ll probably never see the traffic for them in the first place to know you should use them.
• That tool listed in number two? This is an article about it from the company that owns the tool
• Coverage of a search term research session at a conference from 2007
• A compilation of articles on the topic
• A page for marketing terms. Not search term research, just marketing terms
• Another page with coverage of a search term research conference session in 2007
• A press release about a biospace research project.

Did I say ugh? I’ll say it again. Ugh. It’s self evident how many of these pages are clearly NOT the best on the topic out of the millions of pages that Bing could pick.

For some reason, I see completely different results than this when I use Safari, rather than Firefox:

These are generally better, but I still get weird outliers like one for a Utah History center or a place to buy essays.

Yahoo: 50% Irrelevant

How about Yahoo? Ugh again:

Pages for biotech research, autism research, research at Oregon Health & Science University plus two for Lexis/Nexis show up. That’s 50% of the top results completely off target for what I searched for. Not just 50% so-so results. They’re just totally not right. At all.

Ask’s OK, If You Can Stand The Ads

Ironically, given I’ve written it off as a serious search engine last year, Ask seems to have fairly decent results for the topic. There’s a bad press release of someone speaking on the topic, but everything else is good or at least related to the subject.

Of course, you have a giant ad unit containing five paid listings that’s shoved between the first result and the rest. Then another five at the bottom. Then one more paid listing after that, with no disclaimer as required by the Federal Trade Commission.

Things Feel Worse, But Hard To Quantify

Sadly, we still have no commonly accepted measurements of relevancy across search engines, and it’s an area that gets harder and harder to assess, as more material is blended in alongside web page results. That’s something I’d still like the search engine to collaborate on, some independent regular assessment of their quality.

To me, it feels like they’re getting worse, not better. But I can’t document that. What I can do is demonstrate without much difficulty, for areas where I have subject expertise, how bad they can be. They get by because along with the bad, there’s enough good. But they should be better than this.

Google’s Marissa Mayer has announced at the Web 2.0 Summit that Google Social Search will be launching in the coming weeks. I’ve seen an early release of it. It’s way cool. Below, what details we have now about this plus some follow up on today’s Google-Twitter search deal that was announced.

NOTE: Google Social Search is now live. See our Google Social Search Launches, Gives Results From Your Trusted “Social Circle” post.

I’m in a bind because I can’t say more about the product than what Mayer released today. I wasn’t able to make it to Web 2.0 nor were her remarks on the product broadcast live. TechCrunch was there and summarizes what she said this way:

There’s a new Google product called Social Search that is launching in Google Labs. This is a new feature that allows you to see results for queries from people in your social network. This works by using your Google Profile. If you fill it out with the other social networks you’re a member of, such as FriendFeed, Google will scan who you are connected to and give your results from those people.

For example, I have a Google Profile here. On that page, I’ve listed my Twitter account. This means when I’m signed into Google, it can tell who I am and what my Twitter account is with certainty. Then when I search, it can offer to show me web pages that are related to other people in my Twitter profile.

More specifically, if I were do to a search relating to journalism matters, because I follow a number of people in the journalism field (not everyone might see this Twitter List yet), I’d get back both “regular” search results as well as those that are from people who I follow. News.com notes that Mayer said these would appear at the bottom of regular search pages.

Other links from social sites such as Facebook or LinkedIn could also be added to your profile (any link can be added to it). To the degree Google can see your network, those can be used to filter your results.

Our Search 4.0: Social Search Engines & Putting Humans Back In Search article from last year talks more about how search results potentially can be influenced by your friends in general, plus it revisits the personalization that Google already does based on your own behavior. In some ways, the new social search that’s coming is like personalized search extended to tap into your friends and followers network.

I wish I could get more specific, but as I said, I’m limited to the details already released (and everything above is based on what’s out there from Google itself). I wish even more that I had some screenshots to show. But when I saw it last month, I joked on Twitter that I had a Scoblesque tear running down my cheek. OK, it wasn’t THAT awesome. But it was pretty impressive. I don’t see a lot of things that make me go “wow,” that’s useful. This did.

The social search product also predates today’s news that Google has a partnership with Twitter to tap into its data. That means Social Search doesn’t depend on the Twitter deal, but it certainly should help.

What exactly will Google do with the Twitter data. Are we getting a dedicated Twitter search engine like Bing Twitter Search launched today?

Google’s kind of cagey on that front. Mayer said at Web 2.0 that it will be integrated in to regular results. But what’s that mean? Integrated only using Universal Search, which could mean there’s also standalone Twitter search engine out there (just as there’s a standalone image search, news search, blog search and so on)?  Integrated to use Twitter data as part of the core ranking data?

I couldn’t get clarity on whether there will be a standalone Twitter search. Personally, I think there will be, or that there will be a combined microblog search service. We know Google has at least gotten people to translate a name for that service.

Whether that type of dedicated search for microblogged content service gets integrated into the completely different Social Search service that refines results on your social network remains to be seen.

Certainly Google sees the microblogged content as something that needs to be gathered and somehow integrated alongside web pages. Johanna Wright, director of product management at Google, talked to me today about this.

“There are things on Twitter that you can only find on Twitter,” she said, especially local happenings that might never see an actual news article written about them.

One example Wright gave, of stories she says Google is collecting, was about an art project where 2,000 “invisible dog” leashes were handed out in Manhattan. You know, those solid leashes that look like  you have an invisible dog holding them up? No one wrote a news article about this, but if you were trying to figure out what was happening if you saw people with them, the information was blogged on Twitter.

“Typically, there’s not going to be a news story on this,” Wright said. “So most people wouldn’t know why everyone around them has invisible dogs. This is the kind of information that can be found on Twitter.”

I agree. Our What Is Real Time Search? Definitions & Players article goes into more depth about microblogged content and its value to regular search. But there’s also the other major aspect, mining data of URLs that people are tweeting. Is Google planning much there?

“I think you can use them both. One thing to highlight is the background signal from Twitter, whether or not a story is breaking at Google, we have a number of signals that will tell us when a story is fresh,” Wright said.

Take That, Twitter: Google Hot Trends Integrated Into Google Search is another article from us that covers a primary signal that Google has if something is a hot topic — actual searches on Google that happen. And while Google’s has a “query deserved freshness” algorithm that can very quickly find new pages and rank them in top results, Twitter’s data potentially could make that even faster.

Stay tuned for more in the coming weeks.

For related discussion, see news on Techmeme.

Bing new Bing Twitter Search has gone live today, made possible through a new deal cut with Twitter. Below, a close-up look at features in the new service.

Bing Twitter? Bing Facebook? Bing Social Search?

“Bing Twitter Search” is my name for the new service. Bing tells me officially, it’s called “Bing Twitter.” That sounds awkward, two brands next to each other (as opposed to Bing Video or Bing Images, which combine the Bing brand with a generic term).

We also know that a Facebook deal has been approved, as announced today by Microsoft as part of the Bing-Twitter news. But the Facebook data hasn’t yet been implemented on Bing. Will there be a “Bing Facebook Search” service to come? Or is Bing Twitter likely to turn into “Bing Social Search” that combines Twitter and Facebook data?

Microsoft told me they don’t have anything to say on the Facebook front yet. Personally, I’m expecting we will see a Bing Social Search that combines Twitter and Facebook together, if only because today’s blog post from Bing about the new Twitter service is signed by the “Bing Social Search Team.”

Tweets Versus Links

Real time search engines are confusing creatures. Do they show actual tweets and microblogged content that people are putting out? Links that are being shared through microblog services like Twitter and Facebook? Both?

Our What Is Real Time Search? Definitions & Players article from earlier this year take a long look at this issue, and I recommend reading it to understand Bing Twitter Search better.

Bing is trying to do both. When you do a search, you’ll get tweets at the top of the page and shared links at the bottom:

How Fresh Are Your Tweets?

Let’s focus first on the tweets that Bing brings back and shows at the top of the page. Below, a look for a search on kanye west, a popular topic at the moment over rumors he’s died:

Notice the times. When I snapped this, the latest tweet was 1 minute old, followed by three that were 2 minutes old. But over at Twitter Search:

The top three results are only half a minute old. So, Bing’s missing a lot of tweets. But it’s supposed to have the Twitter “firehose” of everything that happens on Twitter virtually as it happens, so what’s up?

Demand & Ranking

Microsoft’s Sean Suchter, General Manager, Search Technology Center, says his company takes the blame. Right now, all tweets are indeed coming into Bing in real time. But then Bing is trying to remove duplicates, filter out adult content and do some other processing. That’s resulting in the delay, but Bing hopes to improve this going forward. It’s a beta, he reminded — and fair enough, especially a beta in its first day.

In the main results, tweets are sorted by date. The most recent tweet shows at the top, and new tweets push that down. That’s how Twitter Search also ranks things, and it’s a spam magnet (see Twitter’s Real Time Spam Problem).

We’ll see if Microsoft changes this order going forward. As said, they do some filtering beyond what Twitter already does. But when you “drill down” into search results, that’s where they hope the results are even more filtered by quality, popularity & usefulness. Let’s look.

Time Ranking Vs. Best Match Ranking

Notice the “more tweets” link below:

This link appears below the “tweet” search results for any search you do on Bing Twitter Search. When you click on it, you can drill further into the results (in contrast, if you were to keep scrolling down on your original search page, you’d see the “Shared Links” section.

Drilling into the results shows you more tweets, sorted by time:

See the arrow? It’s pointing at the two options you have, “Most recent” sort or “Best match” sort. By default, Most Recent is selected.

Suchter said that when you drill into results, filtering goes way up. Bing may try to drop up to 90% of retweets, for example, so that there’s variety in the results rather than everyone saying the same thing over and over.

Then there’s “Best Match” sorting:

Potentially, this should be awesome. But there are problems, as the arrows will highlight. First, though, the ideal world. There are plenty of times when you want “authoritative” tweets, and that’s what Bing promises here:

On that page, you can change the ordering to “Best Match.” Here we arrange Tweets differently. If someone has a lot of followers, his/her Tweet may get ranked higher. If a tweet is exactly the same as other Tweets, it will get ranked lower. For example, I saw a Tweet from ABC News ranked pretty high in the Best Match mode during the “boy in the balloon” fiasco. By the way, you won’t see any of your tweets if you protected or deleted them, and tweets don’t last more than 7 days in our index.

In particular, Suchter said that each tweet’s ranking is influenced by things such as:

• Estimated authority of the person tweeting (such as number of followers they have, how often they’re retweeted)
• Number of retweets a tweet has
• Freshness of the tweet

Now in the search above, what would be authoritative in a Kanye West search? To me, a tweet that makes it clear it’s not true, so we don’t get a repeat of the Jeff Goldblum death rumor (he’s still not dead, either). And look at the screenshot. The first tweet I point at shows the Miami Herald saying he’s not dead. Good job!

Not so fast. Actually, the first tweet is someone retweeting the Miami Herald saying that. Someone with 1/4 of the followers of the Miami Herald. So why isn’t the Miami Herald being shown?

Suchter said the Miami Herald probably should be up there, and that this is something his team is looking at. It’s a beta, remember?

Now look at the second link. Fox News. OK, we can debate how accurate they are. But it’s a news outlet. Good story about the rumor? Well, no. Actually, it’s Fox News trying to cash in on traffic to currently popular topics on Twitter by writing an article about currently popular topics on Twitter.

Look at the third link. Bing really, really falls down hard here. This is misleading spam, someone trying to cash in on the popularity of the search promising a Kanye “RIP” ringtone, whatever that might be.

Now you were to click on that link, Bit.ly itself would stop you with this scary warning:

Nice that Bit.ly does that — but why didn’t Bing do that for you first? After all, a big part of Bing’s announcement today — which drew applause from the audience listening at the Web 2.0 Summit — was that it would open all shortened URLs (those from bit.ly or other URL shortening services) bit.ly URLs, as you can see in the example below:

That wasn’t happening in the Kanye West search. Suchter put it down to a glitch that his team would be investigating.

Shared Links

Remember I mentioned that when you do a search on Bing, the bottom of the page is devoted to a “Shared Links” section. Here’s a close-up:

The goal here is to show you the hot and popular links that are being tweeted on your search topic. Showing top links isn’t unique. Many any other services do this, as described in my What Is Real Time Search? Definitions & Players post. However, none of those players, to my knowledge, get the fast “firehose” of data that Twitter provides.

How’s the ranking done? Each link is evaluated according to a variety of factors, such as:

• How recently is the link
• How many people are retweeting it
• What’s the authority of the people who are retweeting

Microsoft is also following the links and indexing the content of the pages. In addition, it also understands the “main” or “real” or “originating” URL even if different URL shorteners are used. Notice below how the same link from ZDnet is associated in a cluster where two different shortened URLs are used to reference it:

Notice also that for any popular URL, you see up to two people who have tweeted it. Which two? Again, accounts that are deemed relevant based on their authority, people who are retweeting them, freshness and other factors.

For any popular URL, you can drill down to learn even more about it — again, something that’s not unique to Bing’s service, but it’s still pretty cool. Click on the “more tweets” link for any URL:

Then you’ll get a full view of tweets related to that URL. By default, these are sorted in time order (I’m using a different drill down example than the screenshot above, but you should get the point. I had to go with a different page because for some reason, sometimes the drilldown doesn’t work on Bing Twitter Search):

However, you can do the “Best Match” resort. In this case, here’s Best Matches for this article you’re reading now:

Notice how the top results is not either the Search Engine Land Twitter account nor my own personal Twitter account. Instead, these are ranked 2, 3 & 4. Why wouldn’t they higher?

Again, factors include followers, how much your retweeted and how recent the tweet is. Notice that my tweets are older than the person who’s top listed.

URL Lookup

Want a warp speed way to find out what people are tweeting about your pages? Just copy and paste your page’s URL into the Bing Twitter Search query box. Your page should be listed in the top links section, and then you can drill down to see recent tweets or best match ones.

Bing Twitter Search Home Page

You don’t have to search to use Bing Twitter Search. When you arrive at the home page, you’ll see a tag cloud of popular topics:

You can click on any of these to drill down into popular topics. About the only downside to this is listing those topics on Twitter Search itself has driven up spam, people who will tweet misleading information about subjects. It’ll be interesting to see if Bing manages to stamp down on the spam within its own search results.

Below the tag cloud, you’ll see hot links being shared across Bing overall:

The red arrow in the screenshot above points to how each subject has a heading, and by clicking on the heading, you can get even more info on that topic.

Finally, when you do a search, by default new tweets will flow in automatically into the tweets section of results. You can also use the pause button to stop this:

For more, see Bing’s post about the launch and related discussion on Techmeme here.

AllThingsD has reported that Microsoft will announce a deal with Twitter today to gather its real time data. We’re able to confirm that from a source as well and provide some additional details.

The deal will make Bing the first major search engine to have access to Twitter’s “Firehose” of tweets. It’s not exclusive, however. Google potentially could still do a deal, to.

We’re told that:

• The deal will be announced today shortly after Microsoft’s Qi Lu takes the stage at the Web 2.0 conference at 11:30 Pacific Time today. Some sessions are being broadcast live here, and Lu’s might be one of them.
• There will be a standalone Twitter search service offered at Bing, with some ranking technology other than sort by date involved, and that shortened URLs will be expanded. That service should go live today.
• There will be some integration within the regular Bing service itself

Discussions to gather data from Facebook are also continuing, and there’s a chance a deal might be concluded for announcement today.

We’ll update as we learn more. To understand the importance of Twitter and Facebook data to the major search engines, see my What Is Real Time Search? Definitions & Players. It covers what Bing currently does with limited Twitter data it’s able to get now.

See related discussion at Techmeme.

Postscript: Heard back from Google:

At Google we strive to connect people to all the world’s information and this includes social and real-time information. We’re currently exploring new ways to further integrate this type of information beyond what we already offer with services such as Search, News, Profiles, Reader.

Reading between the lines there, my understanding is Google is still talking with Twitter and Facebook itself. Google’s Sergey Brin told me two weeks ago that he seemed positive some type of deal would be struck with the companies. Given the Twitter deal is non-exclusive, I’d expect that’s still likely the case.

Postscript 2: Microsoft has now confirmed the deal on stage at Web 2.0, as expected above. It has also confirmed a deal with Facebook. Financial terms aren’t being disclosed. Microsoft is not saying if there’s a particular time frame associated with it. Will the date influence Bing’s regular results, Lu was asked.

Postscript 3: From the Twitter blog, news on the deal and a strong sense that Google will get a deal eventually:

We hope more working relationships with organizations in the search business will mean even more variety for users.

See our updated posts:

• Live Today: Bing’s Twitter Search Engine
• Live Blog: Qi Lu Speaking At Web 2.0
• Google & Twitter Have A Deal, Too