Google Is A Scraper Site, Says National Association Of Realtors

May 12, 2009 at 2:09pm ET by Matt McGee

With support from the National Association of REALTORS® (NAR), the Indianapolis Metropolitan Board of REALTORS® (MIBOR) has forced some of its members to stop allowing certain MLS listings to be crawled and indexed by Google because Google (and other search engines) is considered a “scraper” site.

This is the latest episode in a long-running battle over who controls home listings that are part of the Multiple Listing Service, or MLS. Some of the affected real estate professionals plan to ask the NAR to change its opinion on search engines at its national convention this week.

Paula Henry is one of about 15 real estate agents affected by the NAR/MIBOR decision. Like many agents/brokers, she shows not only her own listings on her web site, but also listings from other agents and brokers who participate in a data sharing agreement through the Internet Data Exchange (IDX) system. There are strict rules over how listings from other agents may be displayed. In a post on AgentGenius.com, Paula explains the one that has led to Google’s classification as a scraper site:

Section 15.2.2 – participants must protect IDX information from misappropriation by employing reasonable efforts to monitor and prevent ’scraping’ or other unauthorized accessing, reproduction, or use of the BLC database”

(BLC is the Indianapolis version of MLS.)

After apparently getting a complaint from another agent, MIBOR consulted with the NAR and the NAR confirmed that the rule above applies to search engines. On March 27, MIBOR sent Red Door Real Estate (where Paula Henry does business) a cease-and-desist letter detailing two issues:

“…the National Association of REALTORS® (NAR) is in agreement with our interpretation of the policy that the above described practice of ‘indexing your Web site’ as you have called it, is a method of scraping or reproducing the data”
“Under IDX policy … participants have no authority to advertise those listings [from other participants] in any other way, including Internet search engines”

The C&D letter asks Red Door to either use robots.txt (actually referred to as “rebot.txt” in the letter) to block search engines from crawling the listings on their sites that belong to other agents/brokers, or to remove the non-Red Door listings altogether. In her blog post, Paula writes that additional demands were made in April requiring Red Door to remove the MLS number, street address, and other listing data from the Page Titles and Meta Description tags of pages on their web sites.

Not mentioned in all of this is the fact that Google is practically running its own national MLS database, with the same kind of search and sort options (Price, Beds, Baths, Area, etc.) in Google Maps that you’d expect to find on full-fledged real estate sites.

goog-real-estate

Real estate brokers and agents submit their own listings to Google (or they have a third party do it), so the rules about how other agents’ listings are displayed don’t apply. And it’s not just about Google; Yahoo Real Estate also gets listings from agents and brokers, and places them on highly-optimized pages that, in my experience, often rank well for popular real estate search terms.

As for Google being a scraper site, Paula Henry will be one of two real estate agents speaking to the NAR on Thursday. She says they’ll ask the organization to review and change its policy so that Google (and other search engines) is allowed to index all real estate listings from the IDX system, no matter whose web site they appear on.

Matt McGee is the Search Engine Land Assignment Editor, and offers search marketing consulting and training to businesses of all sizes. He blogs at Small Business Search Marketing and HyperlocalBlogger.com.

See more articles by Matt McGee >

Share, Bookmark & Discuss This Article
More:

See more stories like this in the Members Library! Check out the Features: General, Google: Critics, Google: Web Search, Legal: Crawling & Indexing, Top News, Yahoo: Real Estate sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!

10 COMMENTS ON Google Is A Scraper Site, Says National Association Of Realtors

chiropractic, May 12th, 2009 at 4:02 pm ET:
Premium member since 01/2009

Going to be interesting seeing how this one plays out. Real estate listings in local search has got to be a crazy space.

jasoncook, May 12th, 2009 at 5:21 pm ET:

Full Disclosure: I am the Director of Marketing at RealEstate.com and one of our local brokerages under MIBOR jurisdiction has been cited under this policy. Now…having said that…

Not only is this stance inaccurate, but it is detrimental to the very people that real estate organizations are supposed to service – consumers. The internet has empowered us to search out the information we desire and having various competing options has been paramount to that success. In this case, not only are potential buyers adversely affected, but sellers are perhaps the biggest losers in all of this. If I’m selling my house through an agent, I want to know that everything is being done to expose my home to potential suitors. Blocking search engines from picking up this data limits that reach and forces buyers down a much smaller funnel.

Matt McGee, May 12th, 2009 at 6:47 pm ET:
Premium member since 01/2009

I agree 100%, Jason. My wife is an agent, and if she has 5 listings for sale, we don’t care how people find them — one her web site, on someone else’s site, via Google, whatever. The goal is to sell the home, nothing else matters.

Mike Taylor, May 12th, 2009 at 9:44 pm ET:

@jason – I am Paula’s broker and immediately affected by the ruling from MIBOR/NAR. We need to talk! I am gathering some local support to fight MIBOR and would love an opportunity to chat with you. We already have a good group of us affected by this who want to fight. Visit my website, http://www.reddoorindy.com, and contact me if you to talk about this. Thanks.

Liassic, May 13th, 2009 at 5:26 am ET:

It’s quite ironic that everyone is trying really hard to get their data into Google and NAR/MIBOR are trying really hard to keep it out. Pretty naive of them!!!

jasoncook, May 13th, 2009 at 10:13 am ET:

@Matt – it sounds so simple and logical, right? Kind of amazing that we are even having to discuss something so obvious!

@Mike – I will definitely reach out to you. Thanks for the offer!

jrowles, May 13th, 2009 at 2:38 pm ET:

NAR benefits when Realtor.com gets traffic.

Realtor.com gets its data outside of the IDX rules, which NAR dictates.

Therefore, NAR can dictate anti-competitive IDX rules that hobble the SEO capabilities of IDX-powered sites that benefit Realtor.com and, collaterally, other sites that are not subject to IDX display rules, like Google,Yahoo, Trulia, and Zillow.

The irony is that this policy hurts only the members of NAR, and helps only those (Realtor.com included) who are trying to get into the pockets of NAR members.

Considering that they had to back down from the DOJ last year in a settlement over yet another kind of data feed (VOW for “Virtual Office Website”) you really have to wonder what the brain trust at NAR is thinking.

More at http://www.bloodhoundrealty.com/BloodhoundBlog/?p=8358

paulaspeak, May 14th, 2009 at 8:35 pm ET:

There was a new development in this story today. Apparently the National Association’s MLS committee CHANGED their IDX rule, and now permits search engines to index IDX-fed Web pages!

This is excellent progress as it means more brokers/agents will be able to apply basic Web marketing principles and gain ground in the SERPs. We used the Bruce Clay, Inc. blog this week to discuss this in depth and give advice to this SEO-starved industry. See http://www.bruceclay.com/blog/archives/2009/05/jumping_into_th_1.html for the complete scoop. And thanks for bringing this issue to our attention, Matt!

Jay Thompson, May 18th, 2009 at 7:16 pm ET:

“As for Google being a scraper site, Paula Henry will be one of two real estate agents speaking to the NAR on Thursday”

I was the other agent that addressed the NAR MLS Committee last Thursday.

And the Committee voted, unanimously I might add, to change the language of the policy to:

Participants must protect IDX information from unauthorized use. This requirement does not prohibit indexing of IDX sites by search engines.

This change was then approved by the NAR Executive Committee.

And then it went for final passage by the NAR Board of Directors. At that point, one of the Directors (not coincidentally a MIBOR representative) made a motion to table the vote and send the proposed change back to the MLS committee for “further discussion”.

And that’s exactly what happened.

“Further discussion” in this case means it won’t come up again until the committee meets in November — six months from now.

greggster, July 7th, 2009 at 3:40 am ET:

Reminds me of Napter and the DRM madness – the music industry was busy fighting Napter and over some spare change as Apple took their clients from them. As Princess Leia said so well “The more you tighten your grip, Tarkin, the more star systems will slip through your fingers …”

FREE DAILY SEARCH NEWS RECAP!