Is Your Local Pricing Strategy Blocking Search Engine Spiders?

eMarketer’s recent report on Global e-commerce growth showed online sales globally exceeded $1 trillion in 2012. They further indicate that global e-commerce will grow by an additional 19% in 2013, with the Asia-Pacific region surpassing North America in online sales.

This reemphasizes the importance of articles like Andy Atkins-Krüger’s recent article on international pricing strategies and how to manage pricing on a global scale.

This week, I was doing price comparisons for a few products on international sites and noticed that when I tried to go to the Russian or French versions, I was routed back to the US site. It was obvious that they were using some sort of IP detection, and I was curious why.

I emailed their e-commerce teams and found that they have different pricing schemes in each market and have a big problem with consumers bypassing their local market and buying from another country where the price was cheaper, no taxes or simply just not available locally. There were some countries where the potential for fraud was very high, and they were trying to reduce their risk.

To counter this problem, these companies were using a variety of techniques to force the user to buy from the local site or validate their credit card authenticity. While this may work well to force territorial requirements and minimize fraud, it can be a nightmare for organic search.

IP Detection & The Impact On Spiders

The most common method is to simply identify the IP location of the site visitor and serve them predetermined content based on the county and/or city where the visitor has connected to the Internet.

For example, in the image above, I tried to go to a Russian language site, and its server detected that I was coming from the US and sent me to the US site, presenting me with a note saying they had done that. They even told me I could choose my country, but each time I was sent to the US version.

In another case, while I was redirected, the company was allowing Google to enter the Russian version of the site since the site was indexed and had a good snippet. I was curious if other retail sites were using similar tactics.

I visited a number of local versions of popular and niche e-commerce sites and found many were redirecting me. I even changed my user agent to say I was Google and did a second round of visits, but half still routed me back to their US site. Others did let me have the version I wanted when I told them I was Google. Of those that did account for Google, most did not have an exception for Bing.

So,why is this a problem? For the most part, search engine spiders crawl from a specific country, typically the US for Google and Bing. Baidu and Yandex will crawl from China and Russia, respectively. These detection scripts will see the spider’s IP as being in a specific location and route them to the appropriate local version of the site.

For Google, the application would detect the IP location of the US and would route them to English or US centric content, in every case making content in other languages and countries invisible.

There are other cases where the spider may receive an error message, which is typical for JavaScript and Flash detection, but I am seeing it more for IP and language detection. The screen capture below is an example from Google’s SERPs showing the message it received when it tried to visit a Russian version of a site. I have blocked the name and domain of the site to not embarrass the company.

Testing The Defaults & Making Exceptions

In order to truly understand what your servers are doing, you need to test them so you are confident they are serving the right content for all situations, especially to the spiders. Just asking your IT team what is happening is never enough proof of the right settings. As in the case above, they thought everything was working perfectly.

The most reliable test is to have co-workers, partners or even customers visit the different sites from their various locations with different language settings on and off to see what happens in each case.

You should periodically check the default settings of your server, but also any subscription services your company may deploy, such as Akamai’s Global Traffic Management IP Intelligence, Digital Element’s or cyScape’s CountryHawk IP detection solution.

Because your site, as well as these tools, are often updated, it is important to periodically test the redirections and add it to the QA process for any software upgrades or site refresh.

It can happen, not only for IP detection, but for any technology requirement detection scripts for JavaScript, Flash, and I am even seeing some cases with mobile and other platform detection.

Many companies are using responsive design to accommodate for smart phones and tablet screen sizes. If there is not a clear indication it is a mobile device, it will render the pages for a desktop. This can be a challenge to get your mobile optimized content into Google or Bing’s mobile search results.

Develop a best practice of adding the check for the exclusion to your QA process. Create a list of common user agent names for the major search spiders and a exception to allow them to access the page they requested. By allowing them to visit the page they are requesting, you are not redirecting them, and this is not considered cloaking or other forms of black hat optimization.

If you look closely at the screen capture, it did detect the spider was from outside, but rather than redirect or give an exception, it let it in but gave it an error message. Unfortunately, it appears the error message is broken. The error message is not inserting the proper dynamic response message giving a “%1” rather than properly inserting the country information.

 Local Content Designation Challenges

The engines have given SEOs the ability to submit content for local markets via XML site maps, designate sections of the site using Geographical Targeting, and most recently, identify specific location/language pages using the hreflanguage tag/xml protocols. These are great tools and give local and global webmasters a lot of control to identify local content. However, they become worthless if the site has rigid and/or incorrect detection scripts.

The engines are trying to crawl more efficiently and are really cranking down on the number of errors they will tolerate, especially redirects and broken links. If you have used these location-specific tools, especially the XML site maps, but then user server detection scripts to redirect them away from the page, the engines will ultimately stop visiting the site altogether.

Global SEO is only getting more complicated, and those who do it for a living need to think more holistically and ensure that all the possible points of failure related to visiting and indexing your content are regularly checked to ensure you’re not negatively impacting your opportunity to capture the explosive growth in global e-commerce.