Bing Continues With Fake Referrers: What Part Of Stop Don’t They Understand?

For over two years now, Microsoft’s search engine has been generating fake information that can make site owners think they’re getting strange search traffic (including porn traffic), when they are not. It’s time for this to come to an end.

The culprit? Microsoft’s “crawler,” the software that visits web pages across the web that go into the search engine. The crawler is known as MSNBot, from the days when the recently renamed Bing search engine was known as MSN Search. When that crawler gathers a web page, it sometimes leaves behind “referrer” information in a web site’s logs — fake referrer information, that is.

Referrer information? That’s a way that a web browser tells a web site the last page it viewed before coming to the site. A better name would be “referral information,” since that’s effectively what referrer data shows — what web page effectively was the “referral” for someone to find your web site.

For example, imagine someone does a search on Google for movies. In their browser address bar, after doing the search, the URL will look like this:

http://www.google.com/search?q=movies

Now when they click on a listing in the results, they leave Google and go to another page. When arriving at that page, their browser will report the URL that they came from — the URL shown above. And see the parts in bold — “google.com” and “movie?” Web analytics software like Google Analytics use this to tell from the referrer that someone came from Google.com having done a search for the word “movies.” That’s how those who look at web analytics data can tell the words people use to reach their sites.

Microsoft’s crawler isn’t like a normal browser. Crawlers don’t normally report referrer data. MSNbot isn’t doing searches on its own Bing search engine, then following listings to visit new pages. But despite this, it’s acting as if it does.

Back in December 2007, Microsoft admitted to the fake referrer problem and said it was fixed, after site owners scratched their heads wondering since August of that year why they apparently were ranking for porn terms. They weren’t — it was just that Microsoft said they were in the information MSNbot left behind when visiting. And why do this? The explanation was that Microsoft was trying to fight cloaking, where a site owner shows a search engine’s crawler a page that’s different than what a human visitor sees.

Since then, the referrers keep coming up. Search Engine Land news editor Barry Schwartz has regularly documented when crop up. Reports come out of online forums such as WebmasterWorld or Bing’s own forums. Here’s a short rundown of these reports via Barry’s Search Engine Roundtable site:

As you can see, the problem is happening again. For its part, Microsoft gave us this explanation:

Regarding the fake referrers issue, the webmaster team noticed glitches in the system which were resolved on August 20th. If webmasters continue to see issues post this date, we encourage direct feedback to bwmc@microsoft.com or via the webmaster forums at: http://www.bing.com/community/forums/default.aspx?GroupID=11.

What further feedback is needed? Shouldn’t two years be enough to solve whatever problems Microsoft may have that are generating these fake referrers.

Here’s another reason to fix it. Referrers are one way that people can determine search engine marketshare. If site owners see lots of referrers from Google, it seems an important search engine. And if Bing is generating a lot of fake referrers, then it makes itself appear to have more marketshare than it really has.

I DON’T think that’s the reason these fake referrers are happening. But it is an explanation others out there might believe, and a further reason to simply stop this behavior.

Postscript by Barry Schwartz: After much pushing here and other blogs I write at, Microsoft has said they finally fixed the issue. They didn’t explain why there was an issue, but they did say they fixed the issue. More details over here.

Related Topics: Channel: SEO | Microsoft: Bing | Microsoft: Bing SEO | Top News

Sponsored


About The Author: is a Founding Editor of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also serves as Chief Content Officer for Third Door Media, which publishes Search Engine Land and produces the SMX: Search Marketing Expo conference series. He has a personal blog called Daggle (and keeps his disclosures page there). He can be found on Facebook, Google + and microblogs on Twitter as @dannysullivan.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://www.brickmarketing.com nickstamoulis

    Wow, I think it is amazing how Microsoft does not fix an issue for well over 2 years. And it is typical that they gave you a canned answer to submit a request through their webmaster forum…

  • http://www.terryhoward.net Terry Howard

    Seriously, you find it amazing? We are talking about the people whose browser STILL does not support CSS either properly or completely not to mention several other issues like PNG transparency. Pretty sure people have been requesting fixes for that stuff for like 5+ years. It’s pretty much par for the course if you think about it.

  • http://www.brickmarketing.com nickstamoulis

    Hi Terry – Great points…the question is will Microsoft ever really listen to folks and make changes/improvements that are truly needed?

  • http://www.lightbulbinteractive.com davidsculbertson

    I regularly look at the stats for more than dozen websites. Microsoft’s “approach” makes me doubt referrals from Bing/Live/MSN completely. For a couple of clients, the bounce rate for Bing/Live/MSN referrals is higher than 95%. As as SEO practitioner, this makes it impossible to accurately gauge if there is any value to optimizing for Bing/Live/MSN at all.

  • http://mlgoforth mlgoforth

    Thanks for writing about this Danny.
    Bing/Live/MSN/Microsoft or whatever your name is today. You are not a viable search engine when I can’t see the real traffic to my sites from your search engine. 2 years ongoing, apparently you just don’t care. And you wonder why you have an image problem. Wake up.
    I have contacted you Bing/Live/MSN/Microsoft about the fake referrer traffic several times. Via your own forms and in person at a Search Conference, each time being assured it would be fixed. If you can’t fix this how can you really develop a Search Engine that is better than Google. Please Wake UP!

  • http://bryantsmith.com web page designer

    While Google philosophy is do no evil, I’m starting to wonder if Microsoft’s isn’t the exact opposite… I mean, can they do anything honestly and straightforward?

  • http://www.bing.com/community/forums/default.aspx?GroupID=11 brett yount

    Hi,

    Thanks to our webmaster community and followers we are happy to share an update to the referrer issue that you may have heard or read about recently. First of all, we’d like to express our sincere apologies that this referrer issue continued past the August 20th date when we explicitly stated that it was fixed. With the support of many webmasters’ data, our crawling team was able to pinpoint the root cause and deploy a new fix to stop the referrer string in production. This fix has been verified in all of our production beds.

    Thank you for your patience.

    Brett Yount
    Program Manager | Bing Webmaster Center
    LG Consulting at Microsoft
    Webmaster Forums | Webmaster Blog

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide