Google Now Supports Crawling & Indexing Locale-Adaptive Web Pages

Google announced new support for their crawlers with locale-adaptive web pages, web pages that change the content based on the user’s country origin and/or language settings.

For example, web sites/web pages that you visit that automatically change the content on the page based on you coming from France versus the United States or web pages that automatically change the content on the page based on your language settings being set on French versus U.S.-English. Previously, Google didn’t handle that well or actually at all – they would just see the U.S.-English version of the web page.

Now Google will be able to handle such content by sending GoogleBot from different IPs across the world, as well as let it set language settings. Here are the two methods:

• Geo-distributed crawling where Googlebot would start to use IP addresses that appear to be coming from outside the USA, in addition to the current IP addresses that appear to be from the USA that Googlebot currently uses.
• Language-dependent crawling where Googlebot would start to crawl with an Accept-Language HTTP header in the request.

Google still strongly recommends you use different URLs or TLDs based on having content specific for different countries or languages. Google said:

These new configurations do not alter our recommendation to use separate URLs with rel=alternate hreflang annotations for each locale. We continue to support and recommend using separate URLs as they are still the best way for users to interact and share your content, and also to maximize indexing and better ranking of all variants of your content.

So if you are using the rel=alternate hreflang annotations with separate URLs, keep doing so.

This new support is meant to solve a problem Google is having with understanding sites that deploy locale-adaptive techniques. Google said, “new crawling configurations are enabled automatically for pages we detect to be locale-adaptive, you may notice changes in how we crawl and show your site in Google search results without you altering your CMS or server settings.”

For more details on this, see this help document.