Despite the commendable efforts of the parties to improve upon the initial Proposed Settlement, many of the problems previously identified with respect to the original settlement remain in the ASA [Amended Settlement Agreement].  The United States remains committed to working with the parties on the settlement’s scope and content.

The Department Of Justice’s filings argue, effectively, that class action litigation is the wrong mechanism to address the myriad business and copyright questions at issue in the sweeping agreement. It also presents the related question of whether the court, in fact, has the actual authority to approve the ASA in the context of the specific federal statutes raised and adjudicated in the case.

Here’s a excerpt from the government’s papers, which captures the essence of the objections that are then laid out in a technical-legal way throughout the brief. Key parts have been bolded:

Despite this substantial progress, substantial issues remain. Although the United States believes the parties have approached this effort in good faith and the ASA is more circumscribed in its sweep than the original Proposed Settlement, the ASA suffers from the same core problem as the original agreement: it is an attempt to use the class action mechanism to implement forward-looking business arrangements that go far beyond the dispute before the Court in this litigation. As a consequence, the ASA purports to grant legal rights that are difficult to square with the core principle of the Copyright Act that copyright owners generally control whether and how to exploit their works during the term of copyright. Those rights, in turn, confer significant and possibly anticompetitive advantages on a single entity – Google. Under the ASA as proposed, Google would remain the only competitor in the digital marketplace with the rights to distribute and otherwise exploit a vast array of works in multiple formats. Google also would have the exclusive ability to exploit unclaimed works (including so-called “orphan works”1) without risk of liability. The ASA’s pricing mechanisms, though in some respects much improved, also continue to raise antitrust concerns.

We’ll provide more analysis and context later after we’ve had more time to digest the arguments and objections of the US government. Also see this press release from the US Department Of Justice and this AP article on it. For further background on the case and the proposed amended settlement, also see:

• Revised Google Book Settlement Filed & Live Blogging The Press Call
• Google’s Schmidt To Book Settlement Critics: What’s Your Solution?

Postscript From Danny Sullivan: I’m now reading through the filing and effectively doing a fast live blogging of it. Here are sections that stand out to me. I’ve bolded especially interesting parts. From Page 2:

As a consequence, the ASA purports to grant legal rights that are difficult to square with the core principle of the Copyright Act that copyright owners generally control whether and how to exploit their works during the term of copyright.  Those rights, in turn, confer significant and possibly anticompetitive advantages on a single entity – Google. Under the ASA as proposed, Google would remain the only competitor in the digital marketplace with the rights to distribute and otherwise exploit a vast array of works in multiple formats.  Google also would have the exclusive ability to exploit unclaimed works (including so-called “orphan works”1) without risk of liability.

Also from page 2:

The United States recognizes that the parties to the ASA are seeking to use the class action mechanism to overcome legal and structural challenges to the emergence of a robust and diverse marketplace for digital books, including through the adoption of an “opt-out” system to address the complexity of managing millions of third-party copyrights.  Under existing law, copyrighted works typically cannot be exploited in all of the ways the ASA contemplates without the prior permission of the rightsholders.  But for many works, especially out-of-print works, rights clearance may not be possible as a practical matter. Even if the rightsholders can be located, it may not be clear (as between author and publisher, or as among many competing claims to rights in the work) who “owns” or controls the digital licensing of particular works.  This is especially likely where publication predated, and contracts did not anticipate, the digital era.  Finally, there are no major licensing systems in place by which good faith users can efficiently secure permission from, and render payment to, authors, publishers, and the other rightsholders implicated by the ASA.

This is interesting because opt-out is exactly how web indexing works. Google and other major search engines do not gain the explicit permission of rights holders before making a copy of pages in order to index them for searching purposes. Here, the DOJ seems to suggest that opt-out isn’t good enough and also notes that there are no major licensing systems. In the web word, one could argue that the robots exclusion protocol effectively works as a licensing system. Even if an author isn’t explicitly know, Google can still automatically ask the hosting web server for permission.

Search Engines, Permissions & Moving Forward In Copyright Battles is a primer I wrote that explains these issues for, especially comparing web search to book search. In inability for book authors to automatically opt-out is one reason why I suggested in it that Google not scan copyright books without explicit permission.

This is also a good time to note that many in-copyright books have been scanned with permission. And any that you can read via Google Book Search are there because authors or publishers did grant permission. There’s a great deal of confusion on this issue.

From page 3:

These realities make it difficult if not impossible to engage in lawful large-scale book digitization projects, thereby denying the public the full corpus of 20th century books and, perhaps, unknown benefits of future creativity and economic innovation.

Is this the DOJ saying it doesn’t view the scanning Google has done to be lawful? Not quite, as I’ll get into further below.

Also from page 3:

Despite this worthy goal, the United States has reluctantly concluded that use of the class action mechanism in the manner proposed by the ASA is a bridge too far.  The Supreme Court has cautioned that “Rule 23, which must be interpreted with fidelity to the Rules Enabling Act and applied with the interests of absent class members in close view, cannot carry the large load” of restructuring legal regimes in the absence of congressional action – however sensible that restructuring might be.  Amchem Prods., Inc. v. Windsor, 521 U.S. 591, 629 (1997).2  That caution should be heeded here.  Indeed, it applies with particular force because the legal and structural changes the parties seek to accomplish would confer on one entity a level of market dominance that other competitors without access to the ASA’s special rules and procedures will be hard pressed to challenge for the foreseeable future.  For these reasons, the ASA is not the appropriate way to achieve the laudable goals the parties seek.

As I read it, since the settlement involves books from many parties that can’t be found (”orphan works” that are in copyright but where it is difficult or impossible to currently find the rights holders), a class action suit can’t involve them.

From page 4:

At this time, in the view of the United States, the public interest would best be served by direction from the Court encouraging the continuation of settlement discussions between the parties and, if the Court so chooses, guidance as to those aspects of the ASA that need to be addressed.  The United States is committed to working constructively with all stakeholders on the scope and content of an appropriate settlement of this matter, and on legislative or market-based solutions to ensure a robust marketplace for digital works.

I’m at a loss as how this is supposed to magically happen. The DOJ has indicated that it feels many books were illegally scanned. At the same time, it finds the settlement an attempt to a “laudable goal” of getting around that problem. But if you don’t have a class action suit — and you can’t find the rights-holders of some of those books — what then?

It may be that the two main parties who brought the suit against Google, the American Association of Publishers and the Authors Guild, will only be able to strike a settlement for books that have known rights-holders. As for orphan works, potentially those could be excluded — and Google could potentially continue to use them for searchable purposes unless the actual rights holders come forward. Or, potentially the US government might take action against Google over those works. We’ll see what shakes out as this filing is digested.

Indeed, later on page 4 is this:

The United States accepts the proposition that a properly defined and adequately represented class of copyright holders may be able to settle a lawsuit over past conduct by licensing a somewhat broader range of conduct.

That seems to say if all these unknown rights-holders and their works are excluded, then a class action settlement can go forward. You get more of this on page 5:

In previous submissions to this Court, the United States (and other interested parties) discussed the Rule 23 limitations expressed in Amchem, 521 U.S. at 620, 628-29, see, e.g., U.S. SOI at 6-8 (D.E. 720), which suggests that absent class members cannot be adequately represented as to uncertain injuries or rights that are far removed from the facts underlying the complaint.

Many critics of both the original and amended settlement have focused (quite rightly, I’d say) on the fact that it doesn’t actually settle what the case was all about originally — whether scanning books and showing short portions is fair use or not. Instead, the settlement created a new business arrangement that conferred those rights without exploring the legality. On page 6, the DOJ seems to suggest that the court should only rule on the scanning and short display portion — and that on this issue, it could indeed rule to cover those “absent” or “orphan” rights-holders:

The provisions that settle the specific allegations of infringement in the Class Complaint – those that allow Google to scan millions of copyrighted works and to make available small portions of such works in response to search requests – address disputes within the Court’s subject matter jurisdiction.  Those aspects of the ASA are based on specific conduct that falls squarely within the scope of the case made through the pleadings. There are strong arguments that an appropriate set of publisher and author class representatives can adequately represent all members of the class with respect to reaching a settlement as to the uses of their works challenged in the litigation.

In contrast, provisions of the agreement that would allow Google to actually sell full copies of works online don’t seem to make sense, since Google didn’t do anything like this in the first place. It wasn’t sued over this:

The broader aspects of the ASA stand on a somewhat different footing.  There has not been – and simply could not be – any allegation in this litigation that Google has sold full access to works for which it lacks the right to do so, or even that such activity was threatened.  Indeed, selling such access would have been legally indefensible , and thus would have been at odds with Google’s entire pre-settlement book search strategy, which was premised upon staying within colorable “fair use” grounds.  With very good reason, therefore, Google consciously avoided creating precisely the factual predicate that might support the settlement of book- and subscription-selling claims.  The business models that the ASA authorizes therefore relate to activities in which Google never engaged or threatened to engage, and thus claims of copyright infringement that could not have been brought. Although Rule 23 does not require the Court to survey the claims of every class member to determine if they are ripe, there are serious questions about whether a settlement that resolves future claims by absent class members for activities well beyond the facts underlying the complaint can meet the first prong of the Firefighters test.

Remember earlier when I asked if the DOJ was saying that scanning to make something searchable was a copyright violation? This tells me no. This section seems to suggest that the real red flag would have been if Google had reprinted book for sale. THAT would have been legally indefensible.

From page 8:

Here, in contrast, the ASA authorizes future activities beyond the scope of the conduct alleged in the complaint that do not remedy injuries plaintiffs suffered in the past, nor do they seek to prevent future injuries.  Rather, these provisions provide the defendant with benefits it could not have secured either through trial or even through normal private negotiations.

Again, more of the “this settlement goes to far” theme.

From page 9:

The ASA seeks to carve out an exception from the Act’s normal rules and presumptions, which require a rightsholder to affirmatively grant permission for the kinds of uses contemplated by the ASA.  The parties claim that creating an opt-out exception would better serve the purposes of the Constitution’s Copyright Clause by promoting the progress of science and the useful arts.  That, however, is a judgment better suited for legislative consideration, rather than one for courts to make in the context of approving a settlement under Rule 23.

Here, the opt-out discussion is about how the agreement would allow books to be shown or sold unless the authors opted-out (unlike the opt-out of just scanning, that I discussed before). Many critics have wondered why the agreement doesn’t go the other way — allow Google to show or sell only books with explicit permission. All the parties to the agreement have countered with so many orphan works, it’s easier to go opt-out. Then as a new registry is created to hunt down orphan rights-holders, they can choose to opt-out. Certainly switching to opt-in would have made getting some agreement in place much easier.

From page 10, although the DOJ recognizes that opt-out would benefit the public more, that can’t come at the expense of rights-holders:

The United States recognizes that it is the ASA’s broad grant of rights to Google, coupled with the settlement’s opt-out requirements, that allows for the use of the largest possible universe of digital works.  The United States also recognizes that, although Google’s activities are commercially motivated, its business plan would generate numerous public benefits.  The ASA would achieve these benefits, however, in spite of and not in furtherance of the basic premises of the Copyright Act.

From page 12, the DOJ raises the issue that the court seeks to impose a settlement on works on non-US authors who may not be fully represented:

Nonetheless, there are significant numbers of foreign authors from outside Canada, the UK, and Australia whose works were published in one of those countries or registered in the United States, and thus are subject to the ASA, even though the rightsholders may not have been represented by the new associational plaintiffs.  This point is made clear by foreign governments, which object to the settlement.

Further pages get into pricing issues that frankly get beyond me. Suffice to say, the DOJ is worried there are anti-competitive issues involved.

Page 21 comes back to the issue that the settlement grants rights to Google that it wasn’t originally sued over, and which competitors would be hard-pressed to gain:

There is no serious contention that Google’s competitors are likely to obtain comparable rights independently. For example, Amazon – Google’s likely chief rival digital book distributor were the ASA to be approved – began scanning copyright-protected books in 2002, after firstsecuring permission of the works’ rightsholder(s).  To date, Amazon has amassed a library of approximately three million digital titles.  See Amazon.com, Inc. Obj. at 1 (D.E. 206).  This impressive number pales in comparison to the tens of millions of books Google has scanned or is poised to scan if the ASA is approved.  The suggestion that a competitor should follow Google’s lead by copying books en masse without permission in the hope of prompting a class action suit to be settled on terms comparable to the ASA is poor public policy and not something the antitrust laws require a competitor to do.

I love this part. It’s so blunt and straight-forward. Others could scan books just like Google did and show short snippets. If they did, maybe they’d get sued. But even then, there’s no guarantee a settlement would allow them to sell those books in the way Google will be allowed.

On page 22, the DOJ goes to Google’s core search business:

Finally, wholly apart from the new business ventures contemplated by the ASA, Google’s exclusive access to millions and millions of books may well benefit Google’s existing online search business.  Google already holds a relatively dominant market share in that market. That dominance may be further entrenched by its exclusive access to content through the ASA.  Content that can be discovered by only one search engine offers that search engine at least some protection from competition.  This outcome has not been achieved by a technological advance in search or by operation of normal market forces; rather, it is the direct product of scanning millions of books without the copyright holders’ consent and then using Rule 23 to achieve results not otherwise obtainable in the market.

Here, the DOJ’s argument is weaker, I’d say. Amazon or Microsoft could scan books just as Google did to make them searchable. They could have been doing that for years. They might not have because they interpreted fair use differently. Or, in the case of Microsoft, simply because they didn’t think book search was commercially attractive enough, as it said when it backed out of scanning in 2008 (see Microsoft Burns Book Search – Lacks “High Commercial Intent”). The agreement wouldn’t prevent any competitor from scanning for searching purposes.

However, the agreement — in allowing Google to display books scanned — certainly would give it a more compelling book search service. And as previously covered, it would be much harder to nearly impossible for competitors to get those display rights.

On page 23, some DOJ suggestions:

The United States continues to believe that an approvable settlement may be achievable here, for example, by requiring rightsholders to “opt-in” to the settlement or by narrowing both the scope of the plaintiff class and the relief, to better align with the actual dispute underlying the case….

To bullet-point:

• Change from opt-out to opt-in
• Narrow rights granted
• Stick closer to what the case was about: can you scan for searching purposes

Also on that page, there’s a suggestion of a waiting period so as to reduce the number of works that might get used without explicit permission:

The United States believes there would be real value in creating a meaningful waiting period before Google may commercially exploit out-of-print works without the permission of the rightsholder (e.g., two years from the time the title is publicly listed in the Registry).  Such a waiting period, combined with efforts of the Registry to locate rightsholders, may reduce the number of rightsholders whose works would be exploited without their knowledge

Further points also look to narrow the use of orphan works. And that’s it.

Google Blog Search is getting in the game with its announcement today of “Hot Queries,” a new addition that appears in the upper right corner of the Google Blog Search home page. It shows 10 queries that are “happening now” — or, “currently popular” in Google’s words — in Blog Search. What we don’t know is if the list is ranked in actual order, or randomized; and we don’t know how often the Hot Queries list is refreshed.

That’s not the only new addition to the Google Blog Search home page; right below Hot Queries is a section called “Latest Posts,” which shows (and links to) 10 recent posts from “popular blogs.” Google hasn’t given any indication how it chooses which blogs to feature in this space.

Perhaps the most utilitarian new addition to Google Blog Search is RSS and Atom feeds that let users subscribe to any topic or story. Links for those are in the left column.

Lastly, there’s also an iGoogle Gadget that lets users embed the Google Blog Search home page on their iGoogle page.

In an email conversation, Google’s Jeremy Hylton — head of the search quality group in Google’s New York office — explained how the changes impact the “clusters” of stories that appear on the blog search home page.

“We’re doing a better job of choosing the blog posts to include in clusters,” Hylton says. “We’re also working on changes to expand the number of posts we consider for clustering.”

One of the algorithmic changes is aimed at making sure the home page clusters reward the freshest and most authoritative blog content. Within any cluster, Google wants to find the posts that people are talking about the most.

“We have a lot of ranking signals. We’ve been tuning the ranking and clustering algorithms to make better use of those signals. One important changes is that we’re processing new links much faster, so the post that breaks a story and gets a lot of links is more likely to become the lead story.”

The update also includes some minor design changes. Gone are the blue boxes to the left of each cluster, as shown here. The new layout seems more spartan (”streamlined,” in Hylton’s words):

But Google’s changes are more about what powers Blog Search, not what it looks like.

“With [Thursday's] update,” Hylton says, “the whole collection of ranking changes provides a more authoritative set of results. I think it’s really a great set of bloggers talking about current events.”

Hylton says they’ll continue tweaking the algorithm and interface moving forward to improve the overall user experience.

“Company executives who appeared alongside Schmidt at the media briefing said Google was looking at ways to generate money from the surge of social networking activity on the Internet, as well as at ways of integrating microblogging capabilities, such as those popularized by Twitter, into its search product.”

(emphasis is mine)

It’s not in quotes, so we’re left wondering what the exact words and context were … not to mention how Google might “integrate microblogging capabilities” into search. That seems to suggest something more than just offering “Twitter search” or improving its real-time search capabilities.

On ReadWriteWeb, Marshall Kirkpatrick speculates three possibilities: adding microblogs as a search option (like images, video, etc.), using microblogging links to supplement the freshness factor of Google’s algorithm (which I suspect is already happening to some degree), or adding a status update box to its search interface, perhaps encouraging users to answer the question, “What are you searching for?”

Marshall thinks that last option is most likely, but I’m thinking a version of the first idea is most likely: not just adding a microblog search option, but adding real-time/microblog results as a onebox/universal search result. There’s already a Greasemonkey script that does this. But just as Google imports news articles, blog posts, videos, etc., for its blended search results, I could see microblog updates being pulled into Google’s search results. This would make the most sense on hot topics, breaking news, and the more immediate types of queries/information where Google’s traditional results don’t always give the user the information s/he wants.

Then again, I never thought I’d see the day that Google would let users add or remove search results, so take my guess with the proverbial grain of salt.

Update: On the L.A. Times blog, David Sarno shares the exact text of what Marissa Mayer said yesterday about Google and microblogging:

“What’s really happening in Twitter is that there are a lot of clues in it in terms of what’s happening that’s interesting overall. It’s similar to what we see in Google Trends, where people will often type what they’re interested in into the search box, and we can make some predictions off of that.

So we are interested in being able to offer, for example, micro-blogging and micro-messaging in our search. Particularly in Blog Search and possibly in Web Search, but we don’t have any particular plans to announce.”

So, taking that at face value, it sounds like we shouldn’t expect Google to be adding any update-your-status boxes, but a more obvious integration of Twitter/microblogging content into search results is on the table.

Postscript by Barry Schwartz: Google Operating System spotted that in the Google in Your Language project, Google is asking translators to help translate the description of a possible Google Microblogging search engine. The specific text Google is asking to be translated is:

Recent updates about QUERY. This is the MicroBlogsearch Universal result group header text. A Microblog is a blog with very short entries. Twitter is the popular service associated with this format.

Just some more evidence that Google is working towards a Google Micro Blog search engine.

As I reported at the Search Engine Roundtable this morning, Jeremy Hylton of Google said Google Blog Search “now drops most or all of the links that occur in the blogroll or in other parts of the page that are just boilerplate.”

This issue dates back to early November when Google Blog Search began indexing full text of the blog post, including, in some cases, the blogroll and boilerplate portions of the site. Over the course of now and then, Google has been working on tweaking these detectors to unique content and pushed out a major update a couple weeks ago and now has updated the link operator (i.e. link:www.example.com).

My early tests show very positive results, but if you find any issues, please let Jeremy know in this Google Groups thread.

Back in December, we wrote about Google’s switch to full-text indexing for Blog Search, but that led to problems identifying blogroll links as part of blog posts.

Barry points to a Google Groups discussion, where Google’s Jeremy Hylton says:

“We have launched a ranking change that reduces the number of results that are returned because of blogroll matches. There are still problems to work out, but this change appears to be a big improvement over our earlier fix.

So, it sounds like the problem isn’t fully solved yet, but they’re making progress.

Let’s start, appropriately, with a tweet: comScore posted a note that Twitter’s February traffic was up 55% over January. (We don’t see a formal release of this data yet, but will update when we do.) 55% is nothing to shake a stick at; it’s serious growth.

Steve Rubel dug into Twitter Search traffic (i.e., to search.twitter.com) today, and suggests that it’s about to surpass traffic to Google Blog Search:

Steve’s numbers come from Compete.com, which show Twitter Search getting 1.35 million visitors per month, slightly less than Google Blog Search’s 1.38 million. But other data services don’t show the same thing. Here’s how Quantcast — a service that some believe is generally more accurate — compares Twitter Search and Google Blog Search:

There’s no way to know which is correct without seeing actual data from Twitter and Google (don’t hold your breath…), so it’s anyone’s guess if Twitter Search is close to surpassing Google Blog Search. But from looking at the charts above, it doesn’t seem at all farfetched to think it will happen at some point in the future.

Meanwhile, Hitwise’s Heather Hopkins was also writing about Twitter today, but more specifically about where people go from Twitter.

Somewhat surprisingly (to me, at least) the Hitwise data shows Google as the largest recipient of Twitter traffic … and no, that doesn’t include traffic to YouTube (which I thought would be much higher than seventh):

Another thing worth mentioning is Heather’s comparison of traffic categories. She says Twitter’s clickstream doesn’t send as much traffic to shopping and educational sites (like Wikipedia) as search engines do. It sends more traffic to other social network sites and to personal blogs/web sites than search engines do.

In that sense, Twitter itself behaves just like you’d expect it to — like a social network. But the growth of Twitter Search, as evidenced by the charts and data above, suggests that Twitter is moving into a different realm, somewhere between social networking and search, and perhaps bringing together the best of both worlds. Social search, as Steve Rubel says in his piece today, “adds a much needed layer of trust to traditional search that helps us qualify sources.”

Google is now in this position with the settlement of the Google Book Search Copyright Class Action lawsuit against its book scanning project. As part of the settlement, Google has set up a $125 million fund to settle claims, which is discussed on this dedicated website constructed to report the settlement terms and related rules.

To satisfy the notification requirements, in addition to the settlement website, Google has used a variety of methods to communicate with potential claimants. According to an article in the NY Times, it has used direct mail but it also plans to spend millions ($7 -$8 million) on newspaper and print magazine advertising to alert copyright holders about the settlement. The scope of all this is global — hence the price tag.

I would be remiss if I were not to point out the partial irony here: Google using traditional advertising to promote an online initiative. But the Internet, especially outside the West, still arguably doesn’t have the same reach as traditional media, which are thus critical to communicate with copyright holders, authors and publishers. (As an aside, if newspapers in certain US cities disappear it will be difficult to find a judicially acceptable alternative notification mechanism — perhaps a combination of TV, Internet and direct mail.)

The way the Book Settlement ad spend is apparently being distributed, according to the Times’ article, is 30 percent US, 30 percent “industrialized countries” and 40 percent “rest of world.” This apparently reflects the regional contribution of books to the global library.

The majority of the complaints are that Google Alerts are coming up for irrelevant or outdated blog results. In addition, many blogs are no longer indexed quickly anymore. Plus, Google is indexing blogroll code and including them in the link command results.

Jeremy Hylton from the Google Blog Search team updated the Google Groups thread saying that Google will be conducting “visual experiments early next month” that will start with the link: queries and focus on “blogroll detectors” in the matching algorithm. Personally, I am excited to see improvements with Google Blog Search.

Why do I care? Well, I care for a few reasons. As a searcher, I love being able to find relevant and fresh blogs with content on topics that I am looking for. I know that I am missing key blogs in my discovery process and I can’t wait for the results to become fresher, quicker, and more relevant. Yes, I know, easy for me to say, but hard for an engineer to make happen – good thing I am not a Google engineer.

In addition, I often track who is linking to the articles I write. This way I can see what people are saying about my stories and help clarify my stories when necessary. I often do that by watching Google Blog Search for queries on names such as Barry Schwartz, Search Engine Land, Search Engine Roundtable or RustyBrick. I also use the link operator to find out who is linking to my stories on either Search Engine Land or the Search Engine Roundtable. And ever since November 2008 the link operator in blog search failed for me. Then it got worse when they tried fixing it and got somewhat better with their second attempt towards the end of December.

I know many searchers, webmasters, and SEOs who would love to see future improvements. So I hope this early March update will make for a better Google Blog Search, as Jeremy said it might.

It’s been about two months since Google Blog Search was relaunched with a new front page that summarizes stories. I talked with Google more about some of the inner workings at the end of October and finally am getting around to posting this, spurred by one of the planned changes becoming official. Google Blog Search now uses the full-text of posts (in most cases), rather than using whatever was in a blog’s feed (which could often be only part of a post).

It was always annoying that Google Blog Search only depended on what was put out in a feed, rather than actually indexing the full-text of a blog post. Some publishers don’t put out full-text feeds (like here on Search Engine Land) for a variety of reasons, including the fact that putting out a full-text feed often is misinterpreted by others as a right to reprint a post in its entirety without getting formal permission.

Problem solved! Vanessa Fox’s recent Google Blog Search Changes How It Indexes Posts article covers how Google’s now officially confirmed for the first time that blog posts are being spidered. From what Google posted to its support groups on the subject:

We have changed the way we index blog posts to include the full content of the page. We’ve had occasional complaints about the use of the feed content, particularly the problem with partial feeds that you mentioned. The indexing change has improved the results for a lot of queries, both because we have the full content of the page and because we extract links that are missing from the feeds.

It’s a welcome change. When I talked to Google about it in October, they said it was in the process of being rolled out slowly. Now it is fully live for all new posts that get into Google Blog Search. However, some older posts may not be fully indexed. Google expects that by early 2009, all pages indexed by Google Blog Search since June 2005 — which it said are in the billions — will be fully indexed.

And how do you get in Google Blog Search again? If you’re not already there, the easiest way is to “ping” the service with your blog’s home page or your blog’s feed. You can use this submission form. Better, your blog should automatically ping Google each time a post goes up. Most blogging software is either already enabled to do this or it’s simple to add it. But Google provides further instructions here.

In the past, pinging made Google simply grab the latest post as shown in your feed, which is why those putting out only partial feeds did not have their posts fully indexed. Now, a ping is supposed to cause Google to immediately grab the full text of your post (I haven’t tested this yet, however). If it works as advertised, that means within seconds, your full post should be indexed and searchable within Google Blog Search.

A downside to full-text indexing is something that Barry Schwartz noted earlier, as full-text indexing got rolled out unannounced over the past few weeks. Blogs often have blogrolls, links to other blogs. Now that full-text indexing is being done, links from these blogrolls caused some people to think there were new blog posts being done about them. From what Google also posted in its groups about the issue:

The downside of this change is that we see more results that match only the blogroll and other parts of the page that are common to all of a blog’s posts.

We expected some problems from blogroll matches, but may have underestimated the impact on searches using the link: operator or where the query matches a blog or blogger’s name. We do expect to fix the problem you’re seeing. We’ll use the full page content, but exclude the content that isn’t really part of the post. I’m not sure if we’ll be able to make the change before the end of the year, but we are working on it and are pretty confident that it can be solved. We’ll post an update here when we’ve got a solution.

Google also just sent me this update:

Yes, we’ve got a change that should help with the blogroll issue. Although, it’s not 100% perfect, it performs quite well in our tests. We deployed it for link: queries yesterday and hope to have it deployed for all queries in the next few weeks.

Meanwhile, what about how things are going with the new front page to Google Blog Search? One issue that was plaguing Google Blog Search soon after launch was that spam blogs seemed to be getting featured and sometimes embarrassing play within the service.

Jeremy Hylton, a software engineer who works on the Google Blog Search project, said that Google was aiming to get more reputable blogs featured, that Google Blog Search has an internal ranking of blogs by quality and authority that it could use. Looking today, things seem pretty clean — so I assume they’ve notched up the standards.

A confusing thing to me after Google Blog Search launched was the order story “clusters” or “groups” are listed on the home page as well as the subject-specific pages, such as Technology. Each group has a box that shows you the number of blogs estimated to be discussing a topic and the time period where the discussion is stretching over:

The screenshot above shows how a story about Walmart rumored to be selling iPhones has been mentioned on 41 blogs over 12 hours. Now consider what comes below it:

See how under the iPhone story there’s a another story group about a Google Android phone being released in Korea. How come the iPhone story mentioned on 41 blogs over 12 hours trumps the Android story with 57 blogs mentioning it over 18 hours?

Hylton said that part of the ranking process is to look at the “burst of activity” around a story. For example, a story with a sharp spike in mentions that’s recent compared to other stories might come higher on the list, being deemed newer news.

To see some of the spikes, check out any story’s “cluster” or “group” page. I keep using quotes because Google itself doesn’t have a name for these. But you get to them by clicking on the green link showing the number of blogs talking about a particular story. Here’s one for the Walmart iPhone story that I mentioned. On it, you can see a chart on the left showing the number of blogs covering the story and when those stories were spotted:

As a side note, while each story cluster/group has its own summary page, those URLs sadly don’t seem to remain over time. I had examples from things we discussed back in October like this cluster which now simply resolves to the Google Blog Search home page. I wish the URLs would continue working permanently.

Also related, watch for those story cluster pages to be enhanced over time. Hylton said that Google’s looking at ways they can build out more context about a particular story.