Bing Continues With Fake Referrers: What Part Of Stop Don’t They Understand?

For over two years now, Microsoft’s search engine has been generating fake information that can make site owners think they’re getting strange search traffic (including porn traffic), when they are not. It’s time for this to come to an end.

The culprit? Microsoft’s “crawler,” the software that visits web pages across the web that go into the search engine. The crawler is known as MSNBot, from the days when the recently renamed Bing search engine was known as MSN Search. When that crawler gathers a web page, it sometimes leaves behind “referrer” information in a web site’s logs — fake referrer information, that is.

Referrer information? That’s a way that a web browser tells a web site the last page it viewed before coming to the site. A better name would be “referral information,” since that’s effectively what referrer data shows — what web page effectively was the “referral” for someone to find your web site.

For example, imagine someone does a search on Google for movies. In their browser address bar, after doing the search, the URL will look like this:

https://www.google.com/search?q=movies

Now when they click on a listing in the results, they leave Google and go to another page. When arriving at that page, their browser will report the URL that they came from — the URL shown above. And see the parts in bold — “google.com” and “movie?” Web analytics software like Google Analytics use this to tell from the referrer that someone came from Google.com having done a search for the word “movies.” That’s how those who look at web analytics data can tell the words people use to reach their sites.

Microsoft’s crawler isn’t like a normal browser. Crawlers don’t normally report referrer data. MSNbot isn’t doing searches on its own Bing search engine, then following listings to visit new pages. But despite this, it’s acting as if it does.

Back in December 2007, Microsoft admitted to the fake referrer problem and said it was fixed, after site owners scratched their heads wondering since August of that year why they apparently were ranking for porn terms. They weren’t — it was just that Microsoft said they were in the information MSNbot left behind when visiting. And why do this? The explanation was that Microsoft was trying to fight cloaking, where a site owner shows a search engine’s crawler a page that’s different than what a human visitor sees.

Since then, the referrers keep coming up. Search Engine Land news editor Barry Schwartz has regularly documented when crop up. Reports come out of online forums such as WebmasterWorld or Bing’s own forums. Here’s a short rundown of these reports via Barry’s Search Engine Roundtable site:

• Microsoft Live Search Continues Referral Spam Tests With MSLIVSOP?, January 14, 2008
• MSN Live Sending Odd Referrals — QBHP — to Websites, July 24, 2008
• Microsoft To Fix Fake Referrer Data From Live Search, April 22, 2009
• Microsoft Disables Fake Referral “Feature” Temporarily, April 29, 2009
• More Cloaking Tests In Form Of Fake Referrers From Microsoft Bing?, August 5, 2009
• Bing Classifies Cloaking Detection as “Single Word Query” Issue, August 10, 2009
• Microsoft Claims To Fix Fake Referrals or “Single Word Query” Complaint, August 20, 2009
• Microsoft Bing Still Spamming Fake Referrers In Webmaster Log Files, August 25, 2009

As you can see, the problem is happening again. For its part, Microsoft gave us this explanation:

Regarding the fake referrers issue, the webmaster team noticed glitches in the system which were resolved on August 20th. If webmasters continue to see issues post this date, we encourage direct feedback to [email protected] or via the webmaster forums at: https://www.bing.com/community/forums/default.aspx?GroupID=11.

What further feedback is needed? Shouldn’t two years be enough to solve whatever problems Microsoft may have that are generating these fake referrers.

Here’s another reason to fix it. Referrers are one way that people can determine search engine marketshare. If site owners see lots of referrers from Google, it seems an important search engine. And if Bing is generating a lot of fake referrers, then it makes itself appear to have more marketshare than it really has.

I DON’T think that’s the reason these fake referrers are happening. But it is an explanation others out there might believe, and a further reason to simply stop this behavior.

Postscript by Barry Schwartz: After much pushing here and other blogs I write at, Microsoft has said they finally fixed the issue. They didn’t explain why there was an issue, but they did say they fixed the issue. More details over here.