Microsoft’s “Third Era” Of Search Begins With Departure Of Search Chief Christopher Payne

Microsoft’s top search executive leaving company from the Seattle PI covers
Microsoft’s
Christopher Payne reportedly leaving the company. The AP
also reports
similar news from its sources. Payne took over search efforts at Microsoft back
in 2003. He oversaw what I’d call Microsoft’s "second era" of search, that of
developing a crawler-based search engine to take on Google. Below, some history
on the second era, plus the company’s "first era" of human-powered search and
how a reorganization under the direction of Steve Berkowitz marks a new "third
era" of battling for market share.

First Era: Human-Powered Search (1997-2002)

Search isn’t some relatively new effort that dates back to 2003 at Microsoft.
Search, especially web search, is something the company has seriously pursued
since 1997. In its first era, Microsoft started out with a crawler-based search
engine, one that creates listings by using automation to harvest material from
across the web. It then migrated to building a very good service that relied
primarily on human power, human beings to either catalog the web or customize
top search results with hand-picked answers. Bill Bliss was the person in charge
during most of this period. Here’s how it unfolded, over the years:

1997: "Yukon" Search Rumors Emerge

In August 1997, rumors
emerged that Microsoft was building its own search engine code named
"Yukon." Microsoft went public with its plans in October 1997. Rather than build
its own technology, Microsoft decided instead to lease search from Inktomi.
Searchers were promised
"the freshest, most current index available to consumers."

There was much expectation in some quarters that Microsoft’s entry into the
space would wipe out other players, similar to how Microsoft had succeeded with
software. But I thought this unlikely, given that Microsoft wasn’t doing
anything special. From my
story at
the time:

Microsoft’s entry poses two big questions: will it make searching the web
uniquely better, and related to this, will it adversely affect the major
search engines? The answers seem to be no….

Microsoft is not adding any special search technology to the mix. There is
no killer search app that will be created. This may occur in the future, but
at launch, it will remain more of the same.

Microsoft itself wasn’t even positioning the new service, still without a
formal name, as a search killer. Also from my story then:

"Our goal is not to make MSN.com the number one search site on the
Internet," said Ed Graczyk, MSN’s lead product manager. "Our goal is to make
MSN.com the number one site on the Internet, period. And search is a key
component of that."

1998: MSN Search Debuts

Nearly a year after announcing it would get into search, Microsoft finally
unveiled its own search engine dubbed MSN Search (an idea to call it "Internet
Start" was dumped). As expected, it was powered by Inktomi. It wasn’t
particularly groundbreaking, but Microsoft
promised
that it was still only the first generation of what would come in search:

"This is an evolution of what our search service is going to be," said MSN
product manager Nichole Hardy. "Our ultimate vision is to make MSN.com a place
for people to get things done on the web."

1999: Musical Chairs With Search Providers, MSN Search With Human Power &
Bill Bliss

Microsoft made a big deal about how great Inktomi’s search technology was,
the technology that powered MSN Search. But when a better business opportunity
with AltaVista emerged, MSN dropped Inktomi. What Microsoft
said at
the time:

"The change to AltaVista is a better business deal for MSN. The change is
no reflection on our satisfaction with Inktomi. We had an opportunity to form
a deeper strategic relationship with AltaVista and Compaq. That is the primary
motivation for the change," said Marty Taucher, MSN’s director of network
communications.

Despite plans for a deeper relationship, in December 1999, Inktomi returned
as the primary crawler-based partner at MSN Search. However, it didn’t return as
the most favored provider. That went to LookSmart.

In February 1999, a five-year deal to license the LookSmart human-compiled
directory of web sites was announced. By September, that information
was live
and coming up for most search results, with Inktomi only kicking in more unusual
or obscure queries.

Beyond LookSmart, Microsoft was also doing a huge amount of work having human
editors review top listings and provide query refinement options. When deals to
show other search engines on the main MSN.com home page expired, Microsoft also
committed fully to the new service. From my
review at
the time:

So there you have it — a completely revamped service. It’s one that
Microsoft has so much faith in that as of Oct. 1, it stopped promoting other
search services on the main MSN.com home page. Instead, the search box now
queries MSN Search and only MSN Search.

"Where putting our money where our mouth is and betting the farm on MSN
Search," said Bliss.

The "Bliss" being quoted was Bill Bliss, general manager of MSN Search at the
time. I believe Bill took charge of the search product at the beginning of 1999,
though it might have been before that. The beginning of 1999 is when I remember
talking to him regularly, as he oversaw the growth of human editors that I
always thought did a great job giving MSN relevant results for top queries. The
query-to-listing management tools Microsoft built to help those editors were
amazing.

2000-2002: More Features, More Maturity & Paid Listings Arrive

MSN Search stayed largely stable in 2000 and 2001, mainly
adding new
features including things like direct answers that later dropped from the
service only to be rediscovered as "new." Aside from the editorial listings,
Microsoft got its own paid listings program. That’s right — Microsoft had paid
listings of its own before the current adCenter system. "bCentral Keywords" from
Microsoft
offered self-serve CPM-based paid listings on the site. But
by the end
of 2001, paid listings from Overture (formerly GoTo, later bought by Yahoo and
now Yahoo Search Marketing) took over.

Second Era: Crawler-Based Search (2003-2006)

For years, both Yahoo and Microsoft relied on human-based listings for their
main search results. Upstart Google came along with a crawler-based service that
leveraged how people linked to web sites as a way to produce greatly improved
results. This gave Google a single system capable of finding both the best stuff
from across the web plus matches for those "needle in a haystack" searches.
People noticed and started flocking to Google. Yahoo and Microsoft noticed and
started thinking about shifting to use crawlers themselves. Christopher Payne
led the crawler efforts at Microsoft. How that unfolded, over the years:

2003: The "Build Our Own" Decision & Christopher Payne

In late December 2002, Yahoo came out with the surprise news that it planned
to buy Inktomi — the search engine that powered some of MSN Search. The search
engine shopping kept going, with Yahoo announcing plans in February 2003 to
purchase both AltaVista and AllTheWeb. Microsoft had looked at Inktomi and
decided not to buy. It also passed on the other two players, leaving it without
any strong candidates from which to acquire crawling technology, if that’s what
it wanted to do.

At the time this happened, Bliss was no longer overseeing MSN Search.
Instead, John Krass had taken over at some point toward the end of 2002, if I
recall correctly. Bliss had moved higher at Microsoft, then went to Expedia and
earlier this year jumped
to start-up Klir as chief technology officer.

As for crawling technology, Microsoft did want it — but it embarked on the
mammoth task of building the technology itself. Heading up the project was
Christopher Payne.

Microsoft learns to crawl from The Seattle Times and

Search And Destroy from Fortune, both stories out in May 2005, provide very
good histories of the shift to the "build our own" strategy and how Payne was
charged with it. Krass’s tenure over MSN Search was so short between Bliss and
Payne that Payne really became the second person to head the Microsoft search
challenge.

Microsoft started testing its own MSNBot, started massively started
recruiting for employees and
by
mid-2003 was talking openly about building its own crawler to take on Google and
Yahoo.

As back in 1997, many again assumed that Microsoft would build and conquer,
in the way it built Internet Explorer and took over Netscape’s browser share.
I’ve consistently
warned against this view, and here was
one
example of that and explanation why from 2004:

Some observers see Microsoft’s coming battle as only involving Google and
thus assume there will be a repeat of that other classic battle, Microsoft
versus Netscape.

In reality, the battle will be against both Yahoo and Google, which both
have a sizable share of the search market and their own technology. To some
degree, Microsoft also is taking on AOL, another giant in the search space,
though it doesn’t operate its own technology. Finally, unlike with Netscape,
Microsoft isn’t getting a headstart by licensing technology to build on.

2004: Waiting, Waiting

In 2004, there’s was lots of waiting but no real delivery of the new
Microsoft search engine to the mass public. One big change was that Microsoft

ended its relationship with LookSmart, making MSN Search primarily dependent
on the crawler results powered by Yahoo, which had acquired the Inktomi
technology that Microsoft used.

In July,
there were some design changes made to the MSN Search site, but that
continued to provide results primarily from Yahoo. A "preview" of Microsoft’s
own crawler-based results was also released that finally turned into an official
beta out in
November of that year.

2005: Microsoft Switches To Own Technology

In
February 2005, Microsoft flipped the switch, cutting off the Yahoo results and
relying on its own technology. A massive advertising campaign was also kicked
off. The results weren’t impressive. The service wasn’t a Google killer or a
Yahoo killer. Indeed, many felt it was still well behind being ready for prime
time.

2006: Forget MSN Search, Say Hello To Live Search & Steve Berkowitz

Toward the end of 2005, Microsoft
underwent a major
strategy change to offer online products under a "Live" brand, one that’s been
widely ridiculed. As a result, all the marketing and brand building that had
been done to promote MSN Search was wasted. A Windows Live Search beta came out
in March 2006, with the name later shorted to Live Search and the product itself
completely
replacing MSN Search in September 2006. It was just one more step in MSN as
a brand seeming to be abandoned by Microsoft.

On the paid front, Microsoft’s own ad system adCenter

went live in the US in May 2006. It had targeting features more advanced in
some areas than Google or Yahoo. But even Yahoo’s old creaky ad system —
replaced last month with the new
Yahoo Panama system
— beat Microsoft on the most important feature — traffic. Yahoo has more
search traffic than Microsoft; Google even more. Search traffic means money.
Losing it means losing money. And Microsoft was losing it. Throughout 2006, the
company saw a continued
decline in its search share. All the work to build a new crawler, all the
marketing and effort. None of that seemed to be paying off.

Third Era: Battle For Market Share

I know it sounds weird to say the third era is about the battle for market
share. It’s not like Microsoft hasn’t been battling for that for years. But
there were excuses before, especially during the second era when the new
technology to compete with Google was being developed.
Looking At Microsoft’s
Continued Long Game In Search from me in January looks back at the constant
refrain from Microsoft about search being in the "early days" (as they said in
1997) and to give them more time.

Well, Microsoft has had time. Now it has the tools. It’s well out of the
honeymoon period to deliver something solid. That means this era is about market
share, because there are no longer any excuses to justify further losses.

It remains to be seen who will head up this third era. The stories about
Payne say he’s leaving to launch a start-up company rather than as part of the
reorganization that’s underway. If so, replacing him will still be part of that
reorganization. And overseeing it is former Ask.com chief Steve Berkowitz, who
was hired by
Microsoft to oversee the MSN Online Business Group.

Marketing and business development of Windows Live products, such as search,
fall within Berkowitz’s group. He was fairly quiet after going into Microsoft.
Several months after being there, he was talking much more, as in a New York
Times

profile where he described Microsoft as a "cruise ship" that can be slow to
turn:

“I’m used to being in companies where I am in a rowboat and I stick an oar
in the water to change direction,” said Mr. Berkowitz, who ran the Ask Jeeves
search engine until Microsoft hired him away in April to run its online
services unit. “Now I’m in a cruise ship and I have to call down, ‘Hello,
engine room!’ ” he adds with an echo in his voice. “Sometimes the connections
to the engine room aren’t there.”

Now Berkowitz seems to be not just steering the ship but changing the crew.

Microsoft Shifts Windows Vista Executive to Search from Bloomberg covers how
Brad Goldberg, an executive in the Windows side of the Microsoft house, is being
shifted to oversee marketing for Windows Live Search. Mary Jo Foley at All About
Microsoft writes further
on reorganization rumors and changes. Techmeme has a
good roundup of
others writing on the reorg issue, as well as a round-up
here on Payne’s
departure.

Part of that reorg round-up is a
fresh call from Between The Lines writer Larry Dignan for Microsoft to buy
Yahoo. Well, the two have

talked before, though about Microsoft owning a part of Yahoo, not the entire
thing. As I wrote
about that when news of this came out last year, there are pros and cons:

Bringing [Steve Berkowitz] on was a big sign that what Microsoft has been
trying to do internally hasn’t been working — and so something radical such
as an Ask or Yahoo acquisition might be in order.

The big downside is that such an acquisition would give Microsoft yet
another brand to confuse consumers with. After spending hundreds of millions
of dollars over the years to push MSN, they’ve now shifted things behind
making the stupid Windows Live brand their flagship….

So Microsoft’s already coping with the confusion of two major brands.
Adding in Yahoo further confuses matters, unless they perhaps make a brave,
bold move and put everything behind the brand leader in the space, Yahoo.

An acquisition made tons of sense before Microsoft built its own crawler and
ad system. Now it has both. Getting Yahoo last year might have saved Yahoo from
upgrading its own ad system, since it could have used the new one Microsoft
built. But Yahoo has finished that work, as well. So a merger or acquisition
makes less sense — and the brand confusion potential remains significant. But
stranger things have worked.

As a reminder, last year CEO Terry Semel was pretty bullish on Yahoo’s future
against Microsoft:

“My impartial advice to Microsoft is that you have no chance,” Mr Semel
said. “The search business has been formed.”

Some good news for Microsoft. Compete

reported yesterday of a significant uptick in Live Search’s traffic. Compete
hasn’t been in the ratings space as long as competitors comScore, NetRatings and
Hitwise, however. So the uptick is noted, but it will be more reassuring if we
see it also with one or more other services — and especially continuing over
the coming months.