Google Is A Scraper Site, Says National Association Of Realtors

With support from the National Association of REALTORS® (NAR), the Indianapolis Metropolitan Board of REALTORS® (MIBOR) has forced some of its members to stop allowing certain MLS listings to be crawled and indexed by Google because Google (and other search engines) is considered a “scraper” site.

This is the latest episode in a long-running battle over who controls home listings that are part of the Multiple Listing Service, or MLS. Some of the affected real estate professionals plan to ask the NAR to change its opinion on search engines at its national convention this week.

Paula Henry is one of about 15 real estate agents affected by the NAR/MIBOR decision. Like many agents/brokers, she shows not only her own listings on her web site, but also listings from other agents and brokers who participate in a data sharing agreement through the Internet Data Exchange (IDX) system. There are strict rules over how listings from other agents may be displayed. In a post on, Paula explains the one that has led to Google’s classification as a scraper site:

Section 15.2.2 – participants must protect IDX information from misappropriation by employing reasonable efforts to monitor and prevent ‘scraping’ or other unauthorized accessing, reproduction, or use of the BLC database”

(BLC is the Indianapolis version of MLS.)

After apparently getting a complaint from another agent, MIBOR consulted with the NAR and the NAR confirmed that the rule above applies to search engines. On March 27, MIBOR sent Red Door Real Estate (where Paula Henry does business) a cease-and-desist letter detailing two issues:

  1. “…the National Association of REALTORS® (NAR) is in agreement with our interpretation of the policy that the above described practice of ‘indexing your Web site’ as you have called it, is a method of scraping or reproducing the data”
  2. “Under IDX policy … participants have no authority to advertise those listings [from other participants] in any other way, including Internet search engines”

The C&D letter asks Red Door to either use robots.txt (actually referred to as “rebot.txt” in the letter) to block search engines from crawling the listings on their sites that belong to other agents/brokers, or to remove the non-Red Door listings altogether. In her blog post, Paula writes that additional demands were made in April requiring Red Door to remove the MLS number, street address, and other listing data from the Page Titles and Meta Description tags of pages on their web sites.

Not mentioned in all of this is the fact that Google is practically running its own national MLS database, with the same kind of search and sort options (Price, Beds, Baths, Area, etc.) in Google Maps that you’d expect to find on full-fledged real estate sites.


Real estate brokers and agents submit their own listings to Google (or they have a third party do it), so the rules about how other agents’ listings are displayed don’t apply. And it’s not just about Google; Yahoo Real Estate also gets listings from agents and brokers, and places them on highly-optimized pages that, in my experience, often rank well for popular real estate search terms.

As for Google being a scraper site, Paula Henry will be one of two real estate agents speaking to the NAR on Thursday. She says they’ll ask the organization to review and change its policy so that Google (and other search engines) is allowed to index all real estate listings from the IDX system, no matter whose web site they appear on.

Related Topics: Channel: SEO | Features: General | Google: Critics | Google: Web Search | Legal: Crawling & Indexing | Top News | Yahoo: Real Estate


About The Author: is Editor-In-Chief of Search Engine Land. His news career includes time spent in TV, radio, and print journalism. His web career continues to include a small number of SEO and social media consulting clients, as well as regular speaking engagements at marketing events around the U.S. He recently launched a site dedicated to Google Glass called Glass Almanac and also blogs at Small Business Search Marketing. Matt can be found on Twitter at @MattMcGee and/or on Google Plus. You can read Matt's disclosures on his personal blog.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • chiropractic

    Going to be interesting seeing how this one plays out. Real estate listings in local search has got to be a crazy space.

  • jasoncook

    Full Disclosure: I am the Director of Marketing at and one of our local brokerages under MIBOR jurisdiction has been cited under this policy. Now…having said that…

    Not only is this stance inaccurate, but it is detrimental to the very people that real estate organizations are supposed to service – consumers. The internet has empowered us to search out the information we desire and having various competing options has been paramount to that success. In this case, not only are potential buyers adversely affected, but sellers are perhaps the biggest losers in all of this. If I’m selling my house through an agent, I want to know that everything is being done to expose my home to potential suitors. Blocking search engines from picking up this data limits that reach and forces buyers down a much smaller funnel.

  • Matt McGee

    I agree 100%, Jason. My wife is an agent, and if she has 5 listings for sale, we don’t care how people find them — one her web site, on someone else’s site, via Google, whatever. The goal is to sell the home, nothing else matters.

  • Mike Taylor

    @jason – I am Paula’s broker and immediately affected by the ruling from MIBOR/NAR. We need to talk! I am gathering some local support to fight MIBOR and would love an opportunity to chat with you. We already have a good group of us affected by this who want to fight. Visit my website,, and contact me if you to talk about this. Thanks.

  • Liassic

    It’s quite ironic that everyone is trying really hard to get their data into Google and NAR/MIBOR are trying really hard to keep it out. Pretty naive of them!!!

  • jasoncook

    @Matt – it sounds so simple and logical, right? Kind of amazing that we are even having to discuss something so obvious!

    @Mike – I will definitely reach out to you. Thanks for the offer!

  • jrowles

    NAR benefits when gets traffic. gets its data outside of the IDX rules, which NAR dictates.

    Therefore, NAR can dictate anti-competitive IDX rules that hobble the SEO capabilities of IDX-powered sites that benefit and, collaterally, other sites that are not subject to IDX display rules, like Google,Yahoo, Trulia, and Zillow.

    The irony is that this policy hurts only the members of NAR, and helps only those ( included) who are trying to get into the pockets of NAR members.

    Considering that they had to back down from the DOJ last year in a settlement over yet another kind of data feed (VOW for “Virtual Office Website”) you really have to wonder what the brain trust at NAR is thinking.

    More at

  • paulaspeak

    There was a new development in this story today. Apparently the National Association’s MLS committee CHANGED their IDX rule, and now permits search engines to index IDX-fed Web pages!

    This is excellent progress as it means more brokers/agents will be able to apply basic Web marketing principles and gain ground in the SERPs. We used the Bruce Clay, Inc. blog this week to discuss this in depth and give advice to this SEO-starved industry. See for the complete scoop. And thanks for bringing this issue to our attention, Matt!

  • Jay Thompson

    “As for Google being a scraper site, Paula Henry will be one of two real estate agents speaking to the NAR on Thursday”

    I was the other agent that addressed the NAR MLS Committee last Thursday.

    And the Committee voted, unanimously I might add, to change the language of the policy to:

    Participants must protect IDX information from unauthorized use. This requirement does not prohibit indexing of IDX sites by search engines.

    This change was then approved by the NAR Executive Committee.

    And then it went for final passage by the NAR Board of Directors. At that point, one of the Directors (not coincidentally a MIBOR representative) made a motion to table the vote and send the proposed change back to the MLS committee for “further discussion”.

    And that’s exactly what happened.

    “Further discussion” in this case means it won’t come up again until the committee meets in November — six months from now.

  • greggster

    Reminds me of Napter and the DRM madness – the music industry was busy fighting Napter and over some spare change as Apple took their clients from them. As Princess Leia said so well “The more you tighten your grip, Tarkin, the more star systems will slip through your fingers …”

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide