Today we informed our partners that we are ending the Live Search Books and Live Search Academic projects and that both sites will be taken down next week. Books and scholarly publications will continue to be integrated into our Search results, but not through separate indexes.

This also means that we are winding down our digitization initiatives, including our library scanning and our in-copyright book programs. We recognize that this decision comes as disappointing news to our partners, the publishing and academic communities, and Live Search users.

Amazing. And disappointing. Microsoft got mileage out of the idea it was working with the Open Content Alliance as the “good” book search partner not encumbered by controversy that the Google Book Search service has encountered. For background on that, see:

• Battle For Books: Evil Google Versus The Altruistic Open Content Alliance, Dec. 2006
• Dissecting Microsoft Slams At Google As Copyright Infringer, March 2007
• Google: As Open As It Wants To Be (i.e., When It’s Convenient), Nov. 2007

While I had wanted to see the two rival book scanning projects cooperate more with each other, it was still nice to have both major companies involved. But apparently Microsoft figures going after the consumer searchers (as with the Live Search Cashback program) means there’s no time for important but less-revenue generating projects like these. Writes Microsoft:

Given the evolution of the Web and our strategy, we believe the next generation of search is about the development of an underlying, sustainable business model for the search engine, consumer, and content partner. For example, this past Wednesday we announced our strategy to focus on verticals with high commercial intent, such as travel, and offer users cash back on their purchases from our advertisers.

Gosh, Google somehow seems to be able to run a sustainable business model and devote some energy and resources into indexing books and scholarly information, even if those generate little to no revenue. They do it in part because they think it’s good business to provide all types of searches, not just those that will earn them money.

In the middle of a search war, I can understand that a “distraction” like book and academic search might seem like something to Microsoft that has to go. However, Microsoft’s not hurting for cash to keep it up, if it wanted. Dropping it makes Google seem less like the evil giant working for its own benefit that Microsoft would hope people view it as.

Oh well. For the record, Live Search Books launched in December 2006 (versus Google Book Search, launched in December 2003). Live Search Academic launched in April 2006 (versus Google Scholar, launched back in November 2004). Both Google programs continue.

OK, so that does that mean? It’s easiest to show you first, rather than tell you…

Microsoft Product Search

Take a look at Microsoft’s Products Search (http://products.live.com/). Brian Smith went into a lot of detail on the Microsoft’s product search last May in eCommerce, Microsoft Style. Microsoft’s Live Product Search allows people to upload product information into their database, but it also crawls the Web, and extracts information about products.

Libra Academic Search

Another example of indexing on the object level from Microsoft Research Asia, Libra Academic Search, is a computer science bibliography search engine. The page “About the academic search” includes links to a number of papers upon object level retrieval, including an earlier technical report version of the Web Object Retrieval paper.

More than Products and Papers

The product search and the paper search are narrow vertical searches that focus upon crawling Web pages, and finding information that fits within those areas. The academic paper search not only tries to find the names of papers, but also authors, conferences, journals, and research communities. The Web Object Retrieval paper focuses upon extracting that information from pages. The goal of the research extends beyond products and papers. As the authors tell us:

We believe object-level Web search is particularly necessary in building vertical Web search engines such as product search, people search, scientific Web search, job search, community search, and so on.

Incorporation of Object Indexing into Live Search

The product search and the academic paper search are useful, but how well would they do as part of the Web search that Microsoft offers? According to a news article from Microsoft, Search Objective Gets a Refined Approach, those searches have already been integrated into Windows Live:

The “vertical” in Object-Level Vertical Search refers to a specific domain, such as academic search or product search, both of which have been incorporated into Windows Live™. The “object” is an item embedded in Web pages or Web databases, such as a product, a person, a paper, or an organization.

The Object-Level Vertical Search Process

The news article also describes the process of extracting and indexing objects in a nice summary:

The first three steps are:

• Web Crawling: to collect relevant information on the Web efficiently
• Classification: Does a page contain information on products, papers, people, or some other desired category?
• Extraction: pulling specific information about the search query from the relevant Web pages. For a product, for instance, that could mean product name, brand, image, description, and price.

In other words, after finding the information, and understanding that it relates to a specific category, they are putting it into a structured format so that, for instance, products can be compared to one another. There’s more to the process, though:

• Integration: Combining the gathered object information into a concise whole. This includes resolving Web-page idiosyncrasies and naming conventions and making sure that similarly named objects are integrated only if they relate to the actual object being sought.
• Ranking: There are two types of ranking. One, static rank, is handled well by the PopRank algorithm. The second, relevance, is trickier, because an object might be popular, but irrelevant to the query at hand. Because the object description is integrated from multiple Web pages, developing a ranking mechanism is a challenge.

As they note in the article, this method could be used for job searches, for restaurant searches, and even for blog searches.

Ranking Objects by Link Analysis, or PopRank

The last item in the list above talks about ranking objects, and discusses two different parts to that ranking. One is a matter of relevance. The other is a query independent ranking, which they refer to as Poprank. They state that ranking objects may be especially difficult because the object descriptions may come from more than one Web page. So, what is this Poprank?

The answer to that question is likely in another Microsoft paper, Object-Level Ranking: Bringing Order to Web Objects (pdf):

Because it is clear that the more popular the objects are, the more likely the user will be interested in them. So a natural question is: could the popularity of Web objects be effectively computed by also applying link analysis techniques? This paper targets to answer this question. Our answer to the question is yes, but quite different technologies are required because of the unique characteristics of object graph.

To see Poprank in action, try out the Libra Academic Search linked to above.

Ranking for Relevance

Another Microsoft paper that provides an overview of this object extraction and indexing process, Object-level Vertical Search (pdf), introduces the concept of relevancy ranking in its last section, but doesn’t go into much detail on the topic.

Our newest paper (pdf), referred to at the top of this post, does explain how Microsoft might use different language models to estimate the relevance between an object and a query.

As noted, last October in BusinessWeek, Microsoft’s Steve Ballmer positioned Google as "transferring the wealth out of the hands of rights holders." In December, Google came under more fire from the Open Content Alliance, of which Microsoft is part of, as trying to create a closed book search system for only its benefit.

Those were only warm-ups to today’s broadside. Tom Rubin, associate general counsel for Microsoft, delivered a speech to the Association Of American Publishers painting Google as a copyright infringer not to mention dastardly evil ad seller. You’ll find the entire speech here. Coverage also comes from the Financial Times, the Associated Press plus also see roundup coverage from Techmeme here and here. Below, I’ll go through the speech with his references to Google and give you my running commentary on the accusations.

Google Creates Nothing

The second reason I’m pleased to be here is because we have much in common. I recognize, of course, that the works that you help create and publish, and the works we create at Microsoft, seem very different. Still, we share a common understanding of the creative process. We both understand the time and commitment it takes to develop the first germ of an idea into a finished work. More importantly, we both understand the risk it involves – that despite all of our best efforts, a book or software product can still fall flat in the market. I suspect we share many of the same values when it comes to preserving incentives for creativity, so that people will continue to invest in creating works of the very highest quality, not just today, but long into the future.

Remaining true to these values is particularly important as content moves online. I think we can all agree that using the Internet to enhance the market for works is a crucial endeavor and that doing so creates tremendous new opportunities to reach customers. However, the reality, as many of you know, is that authors and publishers often find it difficult just to cover their costs, let alone make a profit, in this new online world. At the same time, companies that create no content of their own, and make money solely on the backs of other people’s content, are raking in billions through advertising revenue and IPOs.

This is the first slam. Google is suggested (you’ll see it IS Google later on) as making no content and just cashing in on others. In contrast, Microsoft creates content (IE, software). That’s not entirely so. If software is content, Google makes plenty of it, such as:

• Gmail
• Google Calendar
• Google Desktop
• Google Docs & Spreadsheets
• Google Earth
• Google Talk
• Google Toolbar
• Picasa

Aside from that, assembling information from others IS content. It’s not easy to do, and it is a real benefit to both consumers and content owners. When content czar Lloyd Braun at Yahoo (now no longer there) "fumed" that Yahoo didn’t have its "own" content on Yahoo News about the Discovery space shuttle mission in 2005, I fumed that he was missing the point:

Create your own programming [as a search engine], and people may not trust you’re going to point them elsewhere. Indeed, that type of pointing IS programming and worked to bring people to Google in droves back when search engines became portals and decided they needed to have "channels" and their own content.

Google’s Not Innovative

Microsoft, I’m pleased to say, has chosen the former path. At its soul, Microsoft is an innovation company, and we’ve been working hard for many years to develop innovative technologies that allow readers to experience books online in new and exciting ways.

After this statement, Rubin then goes on to talk about the two main book projects Microsoft has, both of which are reactions to preexisting and long-standing products from Google. If the suggestion is Google’s not innovative – as I take it — the reality is Microsoft is following in footsteps here. For the record, the projects are:

• Live Search Books, launched December 2006 (versus Google Book Search, launched back in December 2003)
• Live Search Academic, launched April 2006 (versus Google Scholar, launched back in November 2004)

Watch For Google Ads In Books!

What I find exciting about all of these initiatives is that they use great advances in technology to dramatically expand access to works, yet in a way that respects copyright. We believe this is the right path and is one that adheres to the three principles I already mentioned. We also think this distinguishes our approach from one that all of you are familiar with: the Google Book Search project.

The stated goal of Google’s Book Search project is to make a copy of every book ever published and bring it within Google’s vast database of indexed content. While Google says that it doesn’t currently intend to place ads next to book search results, Google’s broader business model is straightforward – attract as many users as possible to its site by providing what it considers to be “free” content, then monetize that content by selling ads. I think Pat Schroeder put it best when she said Google has “a hell of a business model – they’re going to take everything you create, for free, and sell advertising around it.”

All of which can be entirely applied to Microsoft’s own projects. They don’t "currently" carry ads either. But I don’t think Microsoft has ever ruled that out. Remember, Microsoft has no problem selling virtual billboards in 3D representations in its mapping world — ads where no ads actually exist. So let’s not pretend that somehow books would be considered off limits, unless we see a pledge like that.

Tricking Libraries & Getting "Several" Publishers To Cooperate

To accomplish its book search goals, Google persuaded several libraries to give it unfettered access to their collections, both copyrighted and public domain works. It also entered into agreements with several publishers to acquire rights to certain of their copyrighted books. Despite such deals, in late 2004 Google basically turned its back on its partners. Concocting a novel “fair use” theory, Google bestowed upon itself the unilateral right to make entire copies of copyrighted books not covered by these publisher agreements without first obtaining the copyright holder’s permission.

"Persuaded" suggests that libraries somehow can’t think for themselves. To my knowledge, nothing prevents these libraries from also working with others. In fact, I believe the University Of California, to name one, is working with both Microsoft and Google.

Before this part of his speech, Rubin talked a lot about Microsoft’s publisher program, the way it gets copyrighted content into its system:

The second source is our Publisher program, under which we receive books still under copyright from publishers with their express permission, either in digital form directly from the publisher, or scanned from hard copy. Participating publishers have access to an online site – or dashboard – that enables them to manage their publications on Live Search Books. They can choose the amount of text that a reader may preview, create click-to-buy links next to their books, edit metadata, and so on. Several major publishers have signed on to the Publisher Program.

Google has a long-standing similar program, and that’s downplayed. Agreements with "several" publishers for "certain" copyrighted books as Rubin describes is a far cry from hundreds (I think perhaps thousands) of publishers that voluntarily have contributed thousands of books to the Google program. But I think a picture is worth a thousands words:

That’s the Google both at the Frankfurt Book Fair that I attended last September. I believe the fair is the largest gathering of book publishers in the world. There was massive building after massive building filled with publishers.

Notice the picket signs? Notice the angry publishers storming the Google booth? Publishers are upset with Google, yes. But some of those same publishers ironically are also partners in the program. And plenty are partners in the program without being angry at Google. Google is actively involved in the publishing community and has far more contributions than Rubin’s speech suggests.

Microsoft Self-Rules On Copyright

Google’s chosen path would no doubt allow it to make more books searchable online more quickly and more cheaply than others, and in the short term this will benefit Google and its users. But the question is, at what long-term cost? In my view, Google has chosen the wrong path for the longer term, because it systematically violates copyright and deprives authors and publishers of an important avenue for monetizing their works.

As a reminder, we have no idea if scanning books to make them searchable (which is different from reprinting them online) is a copyright violation. That’s the point of the current lawsuits against Google. For all we know, what Google is doing is perfectly legal. As for the "deprive" argument, we’ve also yet to see how this has happened. Google simply does NOT reprint books that are in copyright online unless they have permission to do this.

Opt-In, Not Opt-Out

Rather than delve into this arcane legal issue, what we really should be asking is whether it would be possible for Google to provide its Book Search service in a way that respects copyright. The answer to this question is: of course there is. How am I so sure? Well, because we at Microsoft are doing it. And not just Microsoft. We and others are working on search-driven projects that are proceeding with the express permission and support of copyright owners. And then there’s Google’s own Publisher Partner program, which makes book content available online only after obtaining the necessary authorization.

So, what we really have here are two fundamentally different paths. Google takes the position that everything may be freely copied unless the copyright owner notifies Google and tells it to stop. Microsoft and most other companies, by contrast, take the position that they should get the copyright owner’s consent before they copy. The Copyright Act, in our view, supports this approach. It’s hard to see any justification for exempting Google from its requirements.

First, Google’s position is that everything may be freely copied to make a searchable index. That’s different, as I’ve said (and read in-depth here), from putting the actual books online. Rubin at least acknowledges that "book content" is only put online (IE — reprinted) by Google with authorization.

But let’s be clear. In general, Microsoft does exactly what Google does in terms of taking content and indexing it without permission. That’s how most of its search services operate. They index pages unless site owners specifically opt-out. Microsoft doesn’t call up a site owner and say "Hey, we want to spider your pages. OK?" It just does that. And over in Belgium, that group that sued Google over indexing news content? Yeah, don’t forget they went after Microsoft as well.

Books Are Different!

Of course, I’ve long argued that any search engine asks for permission to index content through things like the robots.txt blocking mechanism. Until recently, I also struggled to understand why book publishers think they should be so special. Time after time, I’d look at the site of a book publisher that was upset with Google for "infringing copyright" by indexing books. Those same publishers don’t block Google from spidering their copyright-protected web pages. So why’s it OK to copy a web page for indexing purposes but not a book? What makes my content online somehow free game but more protected just because it’s printed on dead trees?

Search Engines, Permissions & Moving Forward In Copyright Battles is a post I did last year where I finally switched tothe side of book publishers against Google. I felt they were special because unlike online, there’s no automatic way for them to opt-out:

Still, Google shouldn’t be scanning them, not the in copyright books, not without permission. First and foremost, this is because unlike with the web, there’s no automated way to ask permission. I fully support web indexing, but I support it because there’s an easy way to get permission. That’s not the case with books in copyright. Google can’t ask if indexing is OK. Since they can’t ask, I don’t think they should do it.

Similar to with cached pages, I think Google should back down. Google briefly paused scanning once before. I think they should again, say they feel they’re on solid legal ground but again to be a good corporate citizen, they’re putting things on hold until they can either work out an automated way to seek permission or until they negotiate deals.

As you can see, I’m with Rubin here. Moreover, I’m with the idea that if Google hadn’t started with what’s often seen as its typical arrogant "let’s just do it" attitude and instead talked more with publishers, it might not be set up as the copyright bad guy now. Having said that, any search player making such accusations walks a dangerous line, because it’s easy to turn the focus back on them and find similar examples — as we shall see.

Google’s Bad At Copyright Protection

From the perspective of your business, Google’s approach is troubling for another reason. It assumes, in effect, that Google is the only game in town. Google argues that authors and publishers should simply notify Google if they want to preserve their rights in their works. But what if, as is inevitable, other companies around the world start taking the same approach? Should copyright owners be obligated to track down everyone engaging in unauthorized copying in order to preserve their exclusive rights in their works? Presumably, the desire to preserve these rights is why they asserted copyright in the first place. This approach would be absolutely unworkable in practice, which is probably why Congress in enacting the Copyright Act placed the burden on those who want to copy to get the express consent of the copyright owner, rather than the other way around.

In essence, Google is saying to you and to other copyright owners: “Trust us – you’re protected. We’ll keep the digital copies secure, we’ll only show snippets, we won’t harm you, we’ll promote you.” But Google’s track record of protecting copyrights in other parts of its business is weak at best. Anyone who visits YouTube, which Google purchased last year, will immediately recognize that it follows a similar cavalier approach to copyright. Since YouTube’s inception, television companies, movie studios and record labels have all complained that the site knowingly tolerates piracy. In the face of YouTube’s refusal to take any effective action, copyright owners have now been forced to resort to litigation. And Google has yet to come up with a plan to restrain the massive infringements on YouTube.

Google deserve huge slams over the issue with YouTube. At the same time, we also know it inherited much of that mess, rather than created it. Had Microsoft bought YouTube, it would be mum here.

But more important, does Rubin really want people to start searching on Windows Live to see what copyright infringing content the search engine links to? At least in the US, Microsoft — just like Google — isn’t required to pull down links to such content until the content owners themselves raise issues. A big problem here is that copyright law needs to change. But Microsoft is hardly going to be as innocent as it sounds. No search engine will be. But their nature, search engines can link to content that violates copyright without even knowing.

Google Helping Copyright Infringers

Another example is equally disturbing. Microsoft was surprised to learn recently that Google employees have actively encouraged advertisers to build advertising programs around key words referring to pirated software, including pirated Microsoft software. And we weren’t the only victims – Google also encouraged the use of keywords and advertising text referring to illegal copies of music and movies. These actions bolstered websites dedicated to piracy and reportedly netted Google around $800,000 in advertising revenues from just four such pirate sites. These are not the actions of a company that has the interests of copyright owners as one of its priorities.

Ouch! And Google has that coming. My jaw certainly dropped to discover it had some ad account people actively suggesting that some sites accused of offering pirated movies should buy terms like:

• bootleg movie download
• pirated
• download harry potter movie

For balance, this sounds more like some bad reps rather than a Google company policy. But still, Google deserves the slam. Then again, how does Microsoft do on those terms? Let’s do bootleg movie download:

Hmm. Microsoft’s making money off the term. That’s not so bad, if the sites are all legal. Are they? I suspect at least one of them might be iffy. What about pirated software:

I have to chuckle at that ad. Really, it makes me laugh. See, that ad is making Microsoft money on a per click basis. When you click, you end up on a page that is simply more ads from — Google! That page also contains scraped search results for that search, pirated software, from Microsoft’s own search engine. Potentially, that’s a copyright violation. So Microsoft is making money off someone making money off Google with content that possibly is infringed from Microsoft.

Conclusion

Overall, I have to say it’s disappointing seeing Microsoft come out on an attack stance rather than be positive about what it is doing. Google deserves slams, and I wish they’d change to an opt-in policy for copyrighted books. But for me, with perspective, Microsoft comes across as someone trying to play catch-up and willing to be negative to do it. I don’t like that in political campaigns, and I guess I don’t like it any more in the search wars. But most important, it’s a dangerous game to play. The more Microsoft paints itself as some type of pure protector of copyright, the harder it will fall as people find examples where it fails to meet expectations.

The details on Live Search Books Beta:

The U.S. beta launch of Live Search Books is a big step forward in advancing the way people discover information through the integration of content that has been “off-limits

]]> http://searchengineland.com/microsoft-releases-live-search-books-beta-9990/feed 0