Backtracking on a controversial decision earlier this year, the National Association of REALTORS® (NAR) has adopted a new policy that allows real estate professionals to have their sites — including home listings that belong to others — indexed by search engines.

The controversy reached a peak this year when the NAR agreed with a local decision in Indianapolis that said real estate agents couldn’t let Google and other search engines index the property listings on their sites if those listings belonged to other brokers/agents. In March, the Indianapolis board sent a letter to some agents that essentially called search engines “scraper” sites:

“…the National Association of REALTORS® (NAR) is in agreement with our interpretation of the policy that the above described practice of ‘indexing your Web site’ as you have called it, is a method of scraping or reproducing the data”

At their meeting this week, the NAR Board of Directors revised its policy on home listings and search engines to say that participants “are not required to prevent indexing of their Web sites by recognized search engines.”

That Realtor.org link is also interesting for its comments about the development of REALTORS® Property Resource, which many say is essentially a national search engine for property listings.

Is Twitter allowing search engines access to protected tweets or not? Not, Twitter tells me, though the company probably needs to do a bit more to prevent this type of confusion in the future.

The LA Times reported yesterday about a “Twitter hole” that it believed allowed Google special access to protected tweets, tweets made from Twitter accounts where owners have deliberately chosen not to have their tweets be made public.

Not so, said TechCrunch. The so-called protected tweets that the LA Times was finding in Google looked to be those made from before particular account holders locked down their accounts.

I checked with Twitter and got back the official word from their press office:

The TechCrunch article seems to sum up the confusion pretty well. It seems that the LA Times piece references tweets that were public but later the user protected the account, thus all subsequent tweets are private along with the profile. The tweets prior to that time cannot be un-cached.

Google has not been given a key to the castle…so to speak.

I’m good with this answer except for the word I’ve bolded — that formerly public tweets cannot be uncached. That’s incorrect.

Let’s take an example. Let’s assume you started your Twitter account in March. You started tweeting publicly, then in July decided to be private. Twitter doesn’t try to protect any of your past tweets. In fact, it’s pretty clear about this in its help page on the topic:

If you have a public account and you protect it, all updates after the time of protection will be protected. Your profile will only be visible to approved followers, and existing followers will not be affected.

Please note that tweets from protected profiles will not appear in search results. People will still be able to find your account using the Find People search tool but only people you’ve approved to follow your account will be able to see your tweets. Also note that any tweets posted while your profile is private will remain private indefinitely, and tweets posted while your account is public will remain public indefinitely

But Twitter could try to protect those formerly public tweets. As best I can tell, if you lock down an account, Twitter does make ALL tweets (formerly public or not) inaccessible to everyone accept those the account holder has authorized to see them. That includes search engines like Google or a Twitter-specific search engine like Topsy.

Well, if Google can’t get in to tweets after an account has been protected, why does it show some? And why does Twitter say this will happen?

Google seems to rely on the last information for a tweet that it could see. So you tweeted something in March. Google sees the tweet and records it. If in August, you protect your account. Google tries to revisit your tweets as it does with any web page, to make sure it has fresh information. It can’t get to any of your tweets now.

The ones from August, it never saw them, since they were never public — so it doesn’t list them.

That tweet in March? It keeps showing the information from the last time it saw it. And apparently, it will keep doing this for weeks or months.

Google didn’t send me a comment about this (I did ask, and I might get one later today). But that’s just how I know Google works and can see it specifically working with some protected tweets I investigated today.

As for Topsy, they told me:

Topsy only displays tweets that were once public. The refresh button will make them vanish if the account is now private.

Back to Google. Eventually it should update its old copy of the tweet with what it currently shows to non-authorized visitors, a message that says “This person has protected their tweets” (you can see this for millions of people on Google now).

Twitter could speed that process along by explicitly blocking tweets from a protected account with a meta robots tag configured to remove the page from the index entirely and from cached copies being allowed (the NOINDEX, NOARCHIVE commands).

That wouldn’t guarantee that formerly public tweets are all taken private, of course. Once something’s put out on the public web, it’s very difficult to pull it back. But it could help and seems an easy enough change to do.

If you have a protected account, also keep in mind that those who follow you might retweet what you tweet to the world. If you’re that worried, make sure you pick your followers carefully and regularly keep them informed that you don’t want things retweeted. Otherwise, be prepared for your private tweets to leak out.

For more about search and tweets, see my What Is Real Time Search? Definitions & Players post which cover some ways to make use of Google and its search options feature to drill-down into tweets.

Eric Goldman updated us on a case where Yahoo was sued for showing spam and porn pages for a search on a person’s name. Beverly Stayart was upset that when you searched for her name on Yahoo, Yahoo returned some results that were not to her liking. So she sued Yahoo but the court wouldn’t have it.

Goldman explained that the court rejected her suit on three different levels. She sued Yahoo for Lanham Act false endorsement and it was denied by the court, as expected. The reason it was rejected? Goldman sums it up as follows:

The court denies Various’ 230 defense because its association with the banner ad was unclear. Having dismissed the federal Lanham Act claims completely, the court then declines supplemental jurisdiction over the state law claims. The court also rejects Stayart’s guffaw-inducing request for sanctions against the defendants for having the temerity of moving to dismiss her complaint.

It’s not really a surprise or even news that the US Department of Justice (DOJ) confirmed yesterday it was formally investigating the terms of the Google Book Search settlement. This was known as far back as April and essentially confirmed last month by various publications reporting that formal requests (called “civil investigative demands”) had been issued to book publishers by the DOJ.

What happened yesterday was that the DOJ provided procedural notification to the court that the US is exploring the potential anti-trust dimensions of the Google settlement. The so-called “fairness hearing” to finalize the terms of the settlement will happen on October 7 and the US has until September 19 to “present its views in writing.” That’s not a lot of time.

What did the DOJ say in its two paragraph letter notifying the court of its formal investigation? Here’s the body of the letter:

The Sherman Anti-Trust Act is mentioned in the first paragraph of the government’s letter. The Sherman Act (and related subsequent legislation) is intended to promote competition and prevent consolidation of power and control over markets. For example, Microsoft was found in 2000 to have violated the Sherman Anti-Trust Act for too closely tying Internet Explorer to the Windows operating system.

One of the central questions the DOJ is exploring in its review of the Google Book Search settlement is whether Google gains exclusive rights or benefits to which other parties don’t have access. Some legal scholars have argued that Google does gain what amounts to a monopoly over so-called “orphan books,” books that are out of print but not yet in the public domain and where copyright owners are effectively impossible to locate.

Google has repeatedly said that the agreement is non-exclusive but has not discussed that contention in precise detail. Questions in the anti-trust investigation will include how much control Google has over the “corpus” (material scanned), over pricing of book sales and to what degree third parties can gain access to the material. (New York University is holding a conference on the Book Search Settlement for interested parties on October 8, a day after the fairness hearing.)

Unlike the Google-Yahoo paid search deal that Google walked away from to avoid a legal battle with the DOJ, the Book Search Settlement is different. Because of the underlying lawsuit Google cannot simply walk away here. Putting aside some of the less substantive issues (e.g., whether the attorneys fees are too high) as a basic matter there are essentially only two potential outcomes: approval of the existing settlement or modification of selected terms. If I had to bet I would bet that we will see some sort of modification further expanding third party access to the scanned material.

You can read the settlement itself or take a look at the public-facing Google Book Settlement website for rights holders and publishers. For additional context and background here’s our “library” of Google Book Search stories.

The Register reports a Dutch court has ruled against a Dutch web site, where the web site was sued over the automated snippet used in Google for a keyword phrase.

When someone searched for [Zwartepoorte] and [bankrupt], Google showed Miljoenhuizen.nl and the snippet in the Google search results for Miljoenhuizen.nl was:

Complete name: Zwartepoorte Specialiteit: BMW…This company has been declared bankrupt, it has been acquired by the motordealer I have worked for Boat Rialto…

What is the issue? Well, Zwartepoorte did not go bankrupt and was upset when the Google snippet showed they did. So Zwartepoorte sued Miljoenhuizen.nl, not Google for this error, and won. The site owner had to change their web page so Google would remove the snippet.

This is pretty crazy if you ask me. Google algorithmically makes the snippets. No where on the page did the site say the company went bankrupt. In any event, the court required the web site to modify the page so Google would remove the automated snippet.

“The Court argues that it might be true that the website had no control over the functioning of Google but suggests that these questions about the opacity of Google’s functioning should be addressed in a broader context… then concludes that defendant had its own responsibility,” says van Hoboken, a PhD candidate at the Institute for Information Law at the University of Amsterdam.

Hat tip to Blogstorm for spotting this case.

Discussions between Google and the Associated Press about renewing their content licensing deal continue, I assume, but all’s quiet recently on the negotiation front. I want to disrupt that. It would be wrong in this particular case for both parties to reach a deal where “terms are not disclosed.” The future of journalism, as well as Google’s own reputation, deserves for things to open up.

After threatening a lawsuit against Google several years ago, AP won its first licensing deal with Google in 2006. Google was at pains to stress this wasn’t a deal designed to gain the rights to merely list AP stories. It was supposed to cover more “new” and “extensive” uses of AP material.

Bull. Today, you can still search at Google and find that it fails to list just one “originating” AP story in many cases. Sure, Google hosts AP stories, but it wasn’t like Google set out trying for that goal. It was just as happy as pointing outward. The bottom line was that the deal was a nice wrapper to go around getting the AP off Google’s back.

Now the AP’s been on again that without the right deal, it’ll pull content from Google. Technically, how they’ll do that is absurd. Will the AP robots.txt out its entire site? And ensure that all of its members do the same for any AP story? More likely, it would fall back to the lawsuit front.

But what’s the right deal? What’s Google paying the AP now, and what happened in three years that this amount wasn’t enough? Since the terms were never disclosed, the public can’t judge.

And the public needs to judge, especially as newspapers have secret stealth meetings (which involve AP CEO Tom Curley) which involve sessions like:

Journalism Online: Presentation on proposed service to charge for access to newspaper content and to license that content that (sic) online aggregators

(For more on the stealth meeting, also see the Nieman Journalism Lab report and Techmeme).

See, some in the newspaper industry persist in the assumption that merely listing headlines and summarizing stories is a copyright offense. The AP itself wants to charge for this and already has guidelines that make some people think they might be violating fair use laws when they aren’t (see Do Newspapers Owe Google “Fair Share” Fees For Researching Stories?).

In other quarters, we have people like Forbes.com CEO Jim Spanfeller pulling $60 million dollar figures out of the air, about what he thinks Google owes him and suggesting that Google is helping to destroy “one of the core building blocks of our democracy.”

Since so much is at stake here — democracy itself! — it doesn’t seem right that Google and the AP will reach a deal that no one gets the details too. Shouldn’t all journalistic enterprises, be they traditional or not, understand exactly the value Google’s willing to pay to one with a big mouth and a staff of lawyers?

Moreover, shouldn’t Google be taking a stand on behalf of the majority of publishers who I’d argue have no problem with aggregators or search engines listing their content. A small number of publishers with inflated opinions about their importance are pushing for (and winning) concessions to compensate for their dying business models and lack of foresight. Aren’t the wrong people being rewarded?

You have to appreciate the rich irony. I can’t get the AP to talk with me at all (their execs are all busy, I’ve been told). When their CEO attends a stealth meeting, it’s not even the AP that first reports about it despite the AP being so convinced that they originate stories. Instead, the AP story points to an Atlantic blog post that broke the news. And Google, which touts being open any time it is convenient, is being hush-mouthed here.

Google needs to step up. Defend the “fair use” rights it believes it has on its own behalf and those of the greater web ecosystem. Alternatively, negotiate a deal to solve its AP problem but do it openly, so everyone else understands what exactly is being given up.

With support from the National Association of REALTORS® (NAR), the Indianapolis Metropolitan Board of REALTORS® (MIBOR) has forced some of its members to stop allowing certain MLS listings to be crawled and indexed by Google because Google (and other search engines) is considered a “scraper” site.

This is the latest episode in a long-running battle over who controls home listings that are part of the Multiple Listing Service, or MLS. Some of the affected real estate professionals plan to ask the NAR to change its opinion on search engines at its national convention this week.

Paula Henry is one of about 15 real estate agents affected by the NAR/MIBOR decision. Like many agents/brokers, she shows not only her own listings on her web site, but also listings from other agents and brokers who participate in a data sharing agreement through the Internet Data Exchange (IDX) system. There are strict rules over how listings from other agents may be displayed. In a post on AgentGenius.com, Paula explains the one that has led to Google’s classification as a scraper site:

Section 15.2.2 – participants must protect IDX information from misappropriation by employing reasonable efforts to monitor and prevent ’scraping’ or other unauthorized accessing, reproduction, or use of the BLC database”

(BLC is the Indianapolis version of MLS.)

After apparently getting a complaint from another agent, MIBOR consulted with the NAR and the NAR confirmed that the rule above applies to search engines. On March 27, MIBOR sent Red Door Real Estate (where Paula Henry does business) a cease-and-desist letter detailing two issues:

1. “…the National Association of REALTORS® (NAR) is in agreement with our interpretation of the policy that the above described practice of ‘indexing your Web site’ as you have called it, is a method of scraping or reproducing the data”
2. “Under IDX policy … participants have no authority to advertise those listings [from other participants] in any other way, including Internet search engines”

The C&D letter asks Red Door to either use robots.txt (actually referred to as “rebot.txt” in the letter) to block search engines from crawling the listings on their sites that belong to other agents/brokers, or to remove the non-Red Door listings altogether. In her blog post, Paula writes that additional demands were made in April requiring Red Door to remove the MLS number, street address, and other listing data from the Page Titles and Meta Description tags of pages on their web sites.

Not mentioned in all of this is the fact that Google is practically running its own national MLS database, with the same kind of search and sort options (Price, Beds, Baths, Area, etc.) in Google Maps that you’d expect to find on full-fledged real estate sites.

Real estate brokers and agents submit their own listings to Google (or they have a third party do it), so the rules about how other agents’ listings are displayed don’t apply. And it’s not just about Google; Yahoo Real Estate also gets listings from agents and brokers, and places them on highly-optimized pages that, in my experience, often rank well for popular real estate search terms.

As for Google being a scraper site, Paula Henry will be one of two real estate agents speaking to the NAR on Thursday. She says they’ll ask the organization to review and change its policy so that Google (and other search engines) is allowed to index all real estate listings from the IDX system, no matter whose web site they appear on.

Another publisher is complaining that Google isn’t giving them their “fair share” — this time Forbes CEO Jim Spanfeller. Google makes $60 million off the Forbes brand, he claims (with no proof), and boosting “quality publishers” like Forbes would help Eric Schmidt’s web sewage problem. This is from the publication that sells paid links that helps the sewage rise in Google?

I’ve already dissected some of the misconceptions that Spanfeller and other publishers have about Google in my Sorry, Tom Curley: Don’t Expect A Google Ranking Boost For The AP post from last week. This time, I’ll do a counterpoint to points in Spanfeller’s article over at PaidContent.org.

Spanfeller: For some time there have been murmurings about the relative value generated by Google  vs. the parasitical nature of its business model. In short, is Google being disproportionally compensated for what is fundamentally other people’s work?

Counterpoint: Google sends Forbes and other publishers millions of visits for free. Usually smaller publishers complain if for some reason they lose that traffic due to a ranking change. Newspapers and magazine publishers seem unique in being upset that getting all those free visitors simply isn’t enough. Perhaps Google itself isn’t being properly compensated?

Spanfeller: There is a strong case to be made that Google is indeed getting a bigger piece of the pie than it deserves. It certainly feels that way to content-producing companies when the advertising cycle is in a trough (as it is now) and the advertising lifeblood for branded professional journalism seems to be shrinking by the day. But is there substance to this feeling beyond the pain of lower ad dollars?

Counterpoint: Print publications have had 15 years to find a business model in a world that was moving digital. They haven’t gotten there. Google didn’t even exist when they first started having these pains, but it’s their bane now? These publications also seemed more than happy to fritter away the past three years or so of rising online ad revenues without planning for the inevitable downturns in ad cycles.

Spanfeller: Let’s consider some basic issues, all of which have been discussed in the industry for some time. First off, does the last click get too much credit? Just about everyone I talk with these days agrees that it does. Google is by far the biggest winner in this ill-conceived metric, and by selling branded keywords to the highest bidder, the company is, in fact, working hard to maximize it.

Counterpoint: Somehow I suspect the vast majority of people who search for your brand, Forbes, aren’t clicking on paid ads that show up. Most of them instead are almost certainly clicking on your free listing. Isn’t Forbes by far the winner in those last clicks?

How about an illustration:

Here’s the rundown on each item I’ve numbered:

(1) This is the free listing that you have on Google for your own name. Right there at the top, which survey after survey will tell you is where the majority of people click. How much did you pay for that? Right, nothing.

(2) This is someone selling your freaking magazine. You know, the thing that if its circulation rises, you do better?

Now I can agree, this is annoying if you prefer to sell direct. You might not want to have some third-party competing with you on your own name. Usually smart publishers concerned about this ensure that their official affiliates are barred from doing so. If it’s not an official affiliate, then sorry, the law in the US still seems to allow ads to be triggered from branded terms. You just can’t use the brand name in the text (and these folks are trying to get around that with the whole “Frbs” thing.

Maybe Google should make nice with you and other brand holders by disallowing branded terms from triggering ads, despite legal cases. Might be a good PR move. But then again, you’re not the boss of the word “Forbes.” Sorry, you’re not. You have protected trademark rights relating to it. But if someone’s not passing themselves off as you — or if someone else out there has a “Forbes” brand — they have rights too.

Hey, perhaps Google should never allow the word “iPhone” to be a trigger word for ads that aren’t from Apple. Then the next time my iPhone breaks, and I want someone to fix it, Apple can be the sole advertiser listed — even if others might do the job better and cheaper.

Maybe I have an old Forbes magazine collection that I want to sell. Forbes should have the right to prevent me from advertising this fact?

Or hey, you know sometimes people don’t like companies like McDonald’s. We should prevent them from running ads that might point to content explaining their issues with those companies. Let’s make it a big huge brandfest, shall we?

By the way, you really want to poke at Google about branded terms. Pick a better target — complain that there’s an essential unfairness that Google allows brand terms as trigger words except for Google itself.

(3) Hey, you got me. No idea why these folks think running an ad against “Forbes” makes sense. But who exactly were you losing to the Forbes web site that decided instead that Atlanta real estate was for them? This is also a good time to mention that not everyone who enters “Forbes” actually wants to go to your web site (though most of them probably do — and probably do get there, as I said)

(4) That’s you! It’s your own ad, for your newsletters. Now let me understand this. Google’s sending you tons of traffic for free, but that’s not enough — so you’re buying ads at Google. If you’re buying ads on Google, then hopefully you’re making money off that traffic. And if you are, I gotta believe you’re making money off all those free people Google sends to you each day. How much, would you say? Oh, and free SEO tip. Perhaps if you spent a little time ensuring pages in your online store had even the basics of SEO applied, like unique page titles, Google probably would send you more traffic directly there for free.

Spanfeller: For the most part, a marketer’s top-producing search terms incorporate its brand name … where value chains are muddied beyond recognition—is when Google makes marketers buy their own brand name so that their competitors will not … All of the value of “brand advertising” is discounted as consumers take action on that advertising by using their search bars to navigate to the marketer’s web site. I have been told that in many cases over 90% of search spending by large-brand advertisers is targeted at keywords that are themselves (or have incorporated within them) the actual name of the brand.

Counterpoint: I’m sorry, did a horde of heavily armed Googlers break into the Forbes ivory tower and force you at gunpoint to buy your brand name? I don’t think so. And looking today, just who are your competitors showing up that you’re so concerned about?

Let’s stick with real life rather than the theoretical debate:

• How many visitors per day make it to your web site searching for “Forbes” and other terms incorporating that word? Hundreds? Thousands? Hundreds of thousands? You know the answer — let’s have it.
• How many people does it take until you’d feel you were getting a fair shake?
• What do your own search marketers tell you why they are buying ads for your own brand. Is it really just because they worry about competitors, or might it be that because Forbes wants to push particular revenue-generating assets in front of Google’s audience — and also because search marketers have long reported that a paid ad for your name in conjunction with that nice free link Google gives you makes for better results.

By the way, if your brand is really so big, why are all those consumers so stupid as to not simply enter your URL into the address bar of their browser?

Spanfeller: Search is not really all that great at the moment, a comment repeated time and again by much more astute folks then me. This is especially true when looking for high-quality professionally created content. This is not to say that user-generated content or ecommerce options or product specs should not be returned in search results, simply that there is clearly a better way to showcase the different paths an end user might be pursuing. The idea that everyone is forced into trying to “game” the system so that they get their “fair” (or sometimes not so fair) share is testament to how terribly wrong this entire process has become.

For all the discussion about the vaunted search algorithms, is there any consideration for paid journalism over or even separate from ecommerce options or user-generated blogs and the like?

Counterpoint: Yes, let’s go with the anecdotes. I suppose the economy is picking up now, because I’ve heard a bunch of people tell me they’re feeling better about it. Why don’t you cite that in Forbes?

The reality is that we have no dependable stats on whether search has improved or gotten worse. But I do have my suspicions when a publisher who believes he has “high-quality professional created content” tells me that it’s “especially true” that their publications aren’t being well served.

As for that separate place you want, my jaw dropped when I read that. You have it. Google has all that high-quality professionally created content in a super-secret place, but I’ll reveal its location now. It’s called Google News. Only you and about 5,000 or so other publications get in. For free. Special privileges (see Google’s Love For Newspapers & How Little They Appreciate It). Did you send Google the thank you note yet?

As for different paths to showcase different type of content, here’s a newsflash. Google has what’s called universal search, where they actually segment things into groups like news, products and user-generated content (blog). Sort of what you’re wishing for, except that wish came true back in 2007.

Spanfeller: When a colleague in the industry recently famously went to find information about the not-too-distant hostilities in the Gaza Strip, and did not get an actual professionally created journalistic result until the third page, something is wrong.

Counterpoint: Gosh, did that colleague get any searches right? Maybe most of the time? Or are they commonly having this issue?

Spanfeller: At Forbes.com, we have estimated that Google makes roughly $60 million a year directing folks to our site. And by the way, 40 percent of those dollars are derived from the search terms of Forbes, Forbes.com or Forbes Magazine—simple navigation. Seems like a very nice chunk of change for simply being there.

Seriously, $60 million? Based on what? How on earth did you estimate this? Did you examine how many ads run against any search involving “Forbes?” Did you then magically discover which ones actually got clicks? When you did that, how did you figure out the cost of those ads, when each price is calculated in real-time based on what the advertiser is willing to spend as well as their account history. You want to put a figure out there like that, make it more than a throwaway talking point. Publish it in Forbes itself, as an investigative piece. Because that’s what it would take to get a really reliable number, if such a thing could be arrived it at all from outside Google.

Oh, after you calculate that — then use the average price paid per click, multiple that by all the branded terms that send Forbes traffic for free from Google. Then tell me which is higher. My guess? The free traffic far outweighs the paid traffic you think Google has sucked from you.

Spanfeller: If this inequity of support continues along these lines, we will see a continuing destruction of our journalistic enterprises—enterprises that are one of the core building blocks of our democracy.

Counterpoint: Again, print publications where in trouble before we had Google, but now you’ve got scapegoat to blame for more than a decade of failing to grapple with your fundamental business problems. And let’s add a dash of Google undermining democracy to stoke the fires.

By the way, why is it just Google? I mean, Yahoo sends sites a lot of traffic, too. Yahoo News has a bigger audience than Google News. Yahoo operates pretty much the same as Google. And Microsoft wants to do the same. Why don’t you have an Axis Of Evil Search Engines?

Spanfeller: Last year, while addressing the magazine publishers and editors of the MPA at the Google Campus, Eric Schmidt suggested that the web was a “cesspool” and that it was up to the major journalistic brands to clean it up. Well Eric, in a great many ways, Google has helped to create that cesspool, and as such I would hope that it can be part of the solution.

Counterpoint: Let’s all help, starting with Forbes itself. Forbes has long been known in the SEO world for doing things against Google’s webmaster guidelines, which are explicitly meant to help clean that cesspool. To put it bluntly, Forbes has been pissing in the pool. Some light reading for you, documenting how Forbes has sold links or hosted less than high quality content on its web site:

• Google’s PageRank Update Goes After Paid Links?
• Shining a Light on The Dark Arts of Search Engine Optimization
• Update: Forbes Takes Down Their Paid Links
• Forbes.com Gets a Handjob?
• Forbes former “wrongful death attorney” page

But a picture’s worth a thousand words they say:

Look here on your wine page:

Did Forbes have any of its professional journalists review these “resources” before providing anchor text-rich links to them? I’m guessing not, and that’s an issue. But first, another example:

I assume these links are sold, since they’re labeled “Advertisement.” These are “straight links,” as are the ones in the first example. This means they aren’t delivered in JavaScript or in some other way that would prevent them from passing along credit from your quality site to theirs. Effectively, they are a way someone has bought a vote on your site that these two links are relevant for the words “World Currencies” and “Forex.”

Now in the future, if you search on Google for world currencies, and you get this page full of ads from Business.com ranking tops, please don’t whine that Google should trust in the big brands to sort out the cesspool. We in the SEO space know well that some of those same big brands use the authority of their web sites to enrich themselves, rather than to save democracy.

For more, see related discussion on Techmeme.

Postscript: I’m aware that in doing the counterpoints, I probably come off as a Google fanboy. I’m not. I just like balance and accuracy, and I didn’t feel I got either in Spanfeller’s piece.

Google is far from perfect and has plenty of problems, which I’ve also covered (see today’s Why Hasn’t Google Cleared, Fired Or Suspended Accused AdWords Employee? or my Google: Master Of Closing The Loop? as just two examples of this.

Unfortunately, watching the newspaper and magazine publishers attack Google for their woes is like watching a Keystones Cops movie. They bumble about, puff themselves up with arrogance that they occupy some lofty position while simultaneously put their hands out for a Google bailout.

If they’re going to attack Google, then I want an attack that’s organized, that can’t be so easily shot full of holes and which warrants serious attention. Or I want them to stop attacking Google so it can be attacked, when it deserves it, on far more serious issues without such distraction.

More important, I love journalism and want to see the publishers waste less time looking for someone to blame for the industry’s problems and instead finding new solutions. And no, I hardly see Google as the main reason they’re having issues, though given the rhetoric over the past few weeks, it seems to be the chief boogeyman.

Apparently talks between Google and the Associated Press aren’t going well, or so says Forbes today, with AP chief executive Tom Curley threatening to take his content and play elsewhere.

Where that will be is hard to say. Part of the AP’s original issue with Google was that AP’s own member publications would reprint AP material (which they’re allowed to do as members), causing there to be no single source that could benefit from AP traffic.

Google News Now Hosting Wire Stories & Promises Better Variety In Results covers how after a 2006 deal was struck, AP stories started being hosted on Google itself, as AP wanted (and has long hosted similarly at Yahoo). That would have seen to have solved the AP’s concerns, but apparently not.

The AP wants more money, it seems. How much more is uncertain. Terms of the original deal were never disclosed, and the best I’ve ever seen was Google CEO Eric Schmidt saying for the first time in April that it was a “multi-million deal.”

Part of AP’s current plans are to create its own news portal, as both PaidContent.org and AllThingsD have covered. From AllThingsD:

This has been construed in some quarters as a plan to create a search engine or news portal. But it’s really just an attempt to upgrade the AP’s search engine optimization strategy–that is, trying to get its stuff to show up higher on Google’s (GOOG) search results. It will do that via “search pages,” or “topic pages,” which are par for the course in the Web world. Check out this New York Times (NYT) page  on Somali pirates, or this Huffington Post page on newspapers, and you’ll get an idea of where the AP is going.

If the search page plan works, the pages will be generating plenty of page views when people land on them, and it’s possible that the AP will sell ads on that inventory, Kennedy says. But their real function is to shuttle searchers to the original source material from the AP’s members.

I joked on Twitter that if you’re hoping to suck in Google traffic, it’s generally not a good idea to advertise that fact. Reason? As some point, Google will review its listings, recognized that listing 10 “topic” pages from all the publishers out there probably isn’t providing a diverse results set and change up the algorithm.

More important, Google’s web search quality team — which has nothing to do with Google’s business folks — generally does not take well to people suggesting they’re somehow going to own the search results. AP content probably will start ranking well for some things, but if it started showing up Wikipedia-style for everything, people outside the AP would start complaining about favoritism.

That’s what makes the Forbes piece so puzzling. AP chief executive Tom Curley (who the AP told me was “unavailable” to talk; nor after nearly two hours, does anyone else seem available) sounds naive enough to believe he can force Google into a deal that would give AP preferential treatment in regular search results:

Search rankings on Google News give priority to recognizable news brands like the AP. But Google applies no such algorithmic discretion to general searches. The broader search rankings spread AP content out across the Web, says Curley, encouraging misappropriation by other sites. Curley wants Google to “protect content from unauthorized use and pay us for the longtail.” By “longtail,” Curley refers to the thousands of small sites that collectively drive vast herds of traffic using AP content.

I’ve bolded the key part. Look, I don’t know who — if anyone — is doing SEO for the AP or providing them with advice. Judging from page titles alone, like this key press release about its content enforcement initiative that has no title (a key search ranking factor), they could use some help. And from the statement above, Curley either doesn’t understand how Google works or gets it and is purposely misspeaking. Or perhaps he was misinterpreted. If he’s ever available to talk, I’ll ask him myself.

Google News doesn’t give “recognizable news brands” a boost. I’ve never seen them say this, nor have I seen it actually happen in real life. Google News includes large and small news sites and lists a diverse collection of stories. I know lesser-known news sites do well because I run one of those. At times, I can have a headline story that beats the AP or other mainstream outlets in Google News.

As for the web, actually there’s a considerable debate on whether Google HAS given a brand boost recently (see Google’s Vince Update Produces Big Brand Rankings; Google Calls It A Trust “Change”). Certainly Google CEO Eric Schmidt has been stirring up the pot by repeatedly remarking what a “sewer” the internet is and how “brands” will sort it out.

Certainly if Google starts ranking brands better than other content, they’ll have issues. Brands do not equal trust. Enron had a brand; AIG has a brand — being a brand doesn’t mean that you are more trustworthy or deserve an automatic ranking boost. From my perspective, Google’s algorithm has continued to change over the past few years to reward trusted sites. Many brands have sites that Google has decided are trustworthy, but some don’t.

Curley is foolish if he thinks he’ll browbeat Google into somehow changing its algorithm in web search to reward AP as part of this deal. Google’s search quality engineers wouldn’t stand for that, any more than a journalist would stand for a newspaper CEO marching into a newsroom and demanding that certain advertisers get favorable stories written about them.

If the AP wants traffic to its own site, actually hosting content there is a good first step (and something I wrote about back last year, in Hey AP! How About Running A Real News Web Site?). If they’d done that years ago, they’ve have already earned trust in Google’s algorithms. Instead, they’re going to have to earn that from scratch just like everyone — by having content that lots of people point at with links.

Therein lies another challenge they face. People aren’t going to link to your content when you’re threatening them with your own made-up rules of what you consider fair use, as the AP has been doing and recently became more aggressive about.

In the end, perhaps the AP will walk away from Google. It’s always had the option to block off of its content from being listed. The issue remains that its member publications might continue to take AP stories and post them to the web. AP might try to sue Google for listing this content, but I’d expect legally, it would be the member publications that would be held accountable.

For more, see related discussion on Techmeme.

For a longer look from me at recent newspaper-Google issues, see my Google’s Love For Newspapers & How Little They Appreciate It post.

Postscript: Be sure to watch Tom Curley in this interview with Charlie Rose, where both he and Arianna Huffington talk about the future of news:

It’s remarkable on several fronts:

1) The AP believes that it can divide news content into headlines, lead paragraphs and full stories, each of which can be licensed for use. The essence of being listed in any search engine — not just Google and not just Google News — has been to use the title tag (usually also the headline) of a web page.

Does AP really believe it might strike deals to explicitly give search engines permission list their pages using title tags, when this has been commonly done for over a decade and without any serious legal challenge? And will it put the RSS genie back in the bottle, when RSS feeds typically explicitly offer titles and summaries to the public?

2) The AP wants to create a “newsmap” that takes people to the “authorative source” of a story — to show “who broke the news.” The implication here is that only the AP or its members, or other mainstream publications that AP wants to enlist, originate news.

Much news is originated from non-mainstream publications (yes, such as blogs), that the mainstream press works off of. There are plenty of examples where these non-mainstream publications don’t get credit as originating sources. Do they get included in the newsmap? If not, is the AP effectively doing the same type of robbing that it complains happens to it?

3) Curley asks who is going to pay for the “hard work” and Freedom Of Information Act requests and other types of in-depth journalism. That’s a good question, and I have my own concerns about that (as my Blogs & Mainstream Media: We Can & Do Get Along post covers). But again, it also suggests that only the mainstream media is equipped to do this.

They aren’t, nor are they the only source of hard work journalism. They do, however, often get better access than non-mainstream publications based on what they built over the years.

Perhaps the mainstream publications should give up some of their reserved seats in the White House press room to bloggers who cover politics to wider audiences than some newspapers currently have?

Perhaps with that better access, those bloggers might earn even more traffic, more revenue and be able to spend more time on further hard work journalism, not hindered by the fact that their old business models don’t fit the new publishing world.

Perhaps we might even see some online publications win Pulitzers? See my Time For Google To Fund An Online-Only Version Of The Pulitzers? post for further thoughts on this, as well as how when it comes to online news breaking, often various publications work together to undercover a story, rather than it having to be a huge solo effort.

4) I came away left with the feeling that Curley has no real idea what bloggers and online journalists do. There’s a stereotype the AP seems to have formed, and it’s not an attractive one. I imagine the AP view is that we’re all sitting in basements somewhere in our pajamas scanning Google News for stories from the AP that we merrily reprint without adding value or original content to (I’m actually in sweats today; normally if it’s warmer, I’m in shorts. I rarely wear pajamas to my home office, which isn’t in a basement).

I understand the serious business issues they face and the problems with content theft (which has happened to online publications for over a decade, by the way). But I desperately feel like various AP executives need to get out of their offices, get out of the executive meeting rooms at their member publications and instead roll up their sleeves and visit some actual bloggers at online publications in actions. I’ve been in newsrooms. I came from a newsroom. I know the work that goes on there. I don’t think the AP folks have any clue of the real honest hard work that goes out outside the newsroom. They should get that education, if they really want to form an educated business model for the future.

5) Finally, what exactly is the AP getting from its current deal with Google? We’re hearing it wasn’t a fair shake. OK, cough up, AP. Publish the terms. Tell us exactly what you agreed to in the first place. You’re a news organization; you’re making this an issue, so it’s odd we don’t know your current terms, in order to judge them.

By the way, if you are from a newspaper, you might also want to see my Quick Tips For Newspapers & SEO post.

The Associated Press announced an aggressive new policy whereby it is going to try and get more money for newspaper content online or take legal action against “copyright violators” and those that won’t pay. From the AP’s own press release:

The Associated Press Board of Directors today announced it would launch an industry initiative to protect news content from misappropriation online.

AP Chairman Dean Singleton said the news cooperative would work with portals and other partners who properly license content – and would pursue legal and legislative actions against those who don‘t.

“We can no longer stand by and watch others walk off with our work under misguided legal theories,“ Singleton said at the AP annual meeting, in San Diego.

The quote at the end is vaguely reminiscent of the rogue US general in the 1964 Cold War satire Dr. Strangelove who will no longer allow the “commies” to “pollute our precious bolidly fluids.”

What’s unclear in the new AP policy is whether the organization would now want money for what I’ve just done: linking to a piece of AP content and quoting three sentences. It might.

While some of the interviews and discussions surrounding AP’s announcement suggest that the news association is trying to shut down wholesale or substantial copying of its articles without permission or attribution, there’s reason to believe that they will be going after the kind of use in this article as well. If they don’t want money, perhaps they’ll want advance permission to use the content.

Danny has written a very long and detailed discussion on his personal blog of why the AP policy is misguided, especially with regard to search engines (read: Google). Google has also responded on its Public Policy Blog:

We show snippets and links under the doctrine of fair use enshrined in the United States Copyright Act. The fair use doctrine protects transformative uses of content, such as indexing to make it easier to find [pdf]. Even though the Copyright Act does not grant a copyright owner a veto over such uses, it is our policy to allow any rightsholder, in this case newspaper or wire service, to remove their content from our index — all they have to do is ask us or implement simple technical standards such as robots.txt or metatags.

The Fair Use Doctrine says the following:

[F]air use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright. In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include —

(1) the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;

(2) the nature of the copyrighted work;

(3) the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and

(4) the effect of the use upon the potential market for or value of the copyrighted work.

The notion of what is “fair use” is somewhat ambiguous when applied to the Internet. It’s also at the center of this brewing controversy and may eventually get litigated by the AP. This would be a big risk for AP and newspapers more generally because a decision could go against them. To my incomplete knowledge the doctrine has not been tested in a web-related context around linking and excerpting content online. The AP maintains, however, this isn’t a “fair use” issue:

News aggregators and search companies have long asserted that collecting snippets of articles — usually headlines and a sentence or two — is allowed under the legal doctrine of “fair use.” News organizations have been reluctant to test that idea in court, and it is still not clear whether The A.P. is willing to test the fair use doctrine.

“This is not about defining fair use,” said Sue A. Cross, a senior vice president of the group, who added several times during an interview that news organizations want to work with the aggregators, not against them. “There’s a bigger economic issue at stake here that we’re trying to tackle.”

PaidContent’s interview with Dean Singleton sheds some light on the thinking behind the decision and announcement:

“I think our industry has been very timid about protecting our content, probably because we’ve done so well in the past few years that we didn’t recognize that misappropriation is as serious an issue as it is. As we’re now relooking at business models, it’s become clear that we must protect the rights of our content. … We perhaps have been timid about enforcing [those rights]. No more. We own the content but we’ve let those who spend very little, if any, get the most advantage from it.” What can they get from it at this point? “I am very confident that we will develop new models that help us get more.”

Newspapers want to exert more exclusive control over their content because they have lost faith in their ability to make money from advertising (read: search engine traffic) and will soon be trying to charge for access to it. Subscription and micro-payments models are being evaluated across the industry. Added Singleton:

“What’s becoming clear is that for many, many decades, advertising has supported our news mission and it’s not supporting it today. … If we don’t solve these issues, then survival is not assured. And we expect to solve these issues.”

So where do I start?

In fairness to the newspapers and AP there has been a lot of lifting of their content without permission, attribution or payment. Copyright holders do need to vigorously protect their interests or risk losing them. Having said that, the newspapers are partly hiding behind the copyright issue when, in reality, they have failed to compete successfully online. While they do create great content in many instances, they have lagged in innovating around business models and user experiences.

Danny suggests that the newspapers create “their own Hulu.” It’s a good suggestion. Long ago I also suggested that newspapers become news aggregators themselves. Why didn’t the newspapers invent Digg or Yelp or Trulia? Though there were and are some interesting experiments going on across the industry, there’s something in the culture of newspapers, it would appear, that has effectively inhibited them from fully embracing the Internet and new thinking about their content — beyond putting it up online. In fairness, there’s also a resource issue too.

While I acknowledge the right of newspapers to control their content, there’s a way in which this policy flies in the face of the culture of the Internet. It’s consistent with the worldview and approach that has landed newspapers in the predicament they’re in today. On some level it seeks to turn back the clock.

And while it’s true hypothetically that if there never was a Google News or Yahoo News or Topix (itself owned by newspapers) perhaps people would be going directly to newspapers sites to get their news and other local content. Yet the nature of the Internet made these aggregator sites all but inevitable. Why didn’t the newspapers develop them first?

I acknowledge the dire predicament newspapers are in and that they need to “do something.” (I’ve devoted lots of writing on my blog to newspapers.) And honestly I wouldn’t have any magic solution if I were being asked to devise a strategy to “save the industry” at this point. However they should seek to exploit distribution — including mobile and its new opportunity to charge for content — rather than limit it.

There’s also a way in which this newly announced policy is a giant PR blunder for AP and by extension newspapers. They might have consulted with Google and other “aggregators” and negotiated a more online-friendly policy before going public with an approach that feels both hostile and desperate.

___

Postscript: Google’s Eric Schmidt addresses the NAA (newspaper) conference (audio).