Smartphone Bot Case Study: The Google Smartphone Bot On Holiday In Australia

While Google’s new smartphone bot has been announced, it does not appear to be widely deployed yet; at least that is what I said during a recent presentation in Sydney, where I had the pleasure of meeting Alistair Lattimore (Al). Al does CRO, SEO and agency management for Mantra Group, which is the second largest accommodation provider in Australia with over 100 hotels, retreats and resorts throughout Australia and New Zealand.

Al was at the talk in Sydney and let me know that the smartphone bot had visited some of the sites he is responsible for and told me about the initial results. Mantra.com.au is the site we discussed most; iPhones and Androids are now showing the mobile-optimized urls instead of desktop urls in some (not all) searches.

While I am writing this case study up, Al gets most of the credit for his keen observations about the behavior of the bot and its impact on search results. As it may be useful to have a short case study out there, we’ll review the specific case of the smartphone bot, and how its indexing has played out in the Mantra mobile crawling, indexing and ranking.

Keep in mind this is just one site, and other sites may have different experiences with the smartphone bot, but here is what Al and now I have observed thus far.

Smartphone Bot Crawling

As I alluded above, I have done lots of research to see which sites are being affected and which are not. Many of the large sites do not appear to be showing evidence of the smartphone bot yet.

This includes sites like Facebook, Amazon and YouTube, which all still show ‘www’ desktop urls in searches from Androids and iPhones. Either the bot is crawling them and not indexing the mobile content, or perhaps I am not doing the right tests, to generate the mobile results (more on that later).

One of the most interesting aspects of this case study is that Mantra.com.au was crawled by the smartphone bot in early December, according to Al, slightly before the new bot was even announced (sneaky Google!). Al explained that their content has always been crawled quite regularly, and that they did not do anything special like submit a sitemap for the mobile subdomain or set up any special robots instructions, except to disallow content that did not belong in results at all.

The Mantra site is using conditional 302’s to redirect mobile traffic from the desktop pages to their mobile counterparts. Note that 302’s are much less common in traditional SEO, but very common in mobile user-agent detection and redirection, but a 301 would be fine as well.

The mobile optimized urls do not appear to be exact mirrors of each other, though they do match up in the final element of the file name; for instance, a vacation property in Sydney on Bond Street has the following two urls and meta data:

Desktop Url: https://www.mantra.com.au/new-south-wales/sydney-and-surrounds/sydney/accommodation/hotels/mantra-2-bond-street/

Desktop Title Tag: Mantra 2 Bond Street | Sydney Hotels | New South Wales NSW

Desktop Description Tag: The hotel is centrally located close to Sydney attractions such as the Sydney Opera House, Pitt Street Mall, Darling Harbour and The Museum of Contemporary Art.

Mobile Url: https://m.mantra.com.au/buildings/mantra-2-bond-street

Mobile Title Tag: Mantra 2 Bond Street Sydney NSW

Mobile Description Tag: NONE

Interestingly, the desktop title tag and description tags are still present in the mobile ranking, and so far, the only thing that has changed is the replacement of the desktop page url with the mobile page url as shown below.

This is not exactly what Google had promised in the announcement and description of how the bot worked:

Smartphone Bot Mixes Desktop & Mobile Page Info in the SERP

The crawler is relying almost 100% on the 302 redirect without additional signals of affiliation between the desktop page and the corresponding home page. Mantra is linking back to the desktop site with a text link at the bottom of every page, which uses cookies to stick the user to the desktop site, but it is a page-to-homepage link.

This is great, (and obviously good enough for Google) but it might not provide users or engines as much value as a page-to-page link could in the long term. Also, Mantra is not using canonical tags to help associate the mobile versions of the pages to their desktop counterparts.

Al really did a great job maximizing the crawl efficiency on the mobile site by minimizing DUST (duplicate url, same text). Here is what he told me:

“I’ve worked very closely with our in-house developers to minimize unnecessary redirects within the site & every iteration we’re dropping more & more where possible. We’re handling duplicate content using redirects and have currently handled URL casing (force lower case), protocol (can only view a URL in the intended protocol, but need to get that changed to a 301 as its currently a 302) and I’m waiting on another small update to strip trailing slashes via a 301 as well.”

Al also mentioned that he has made sure that the developers were eliminating unnecessary redirects and keeping the mobile code clean and light. He wins the gold medal here because this will all really ensure that all of his mobile and desktop content will be crawled and indexed without slowing the bots down – which is exactly what has happened.

Mantra Group is also using user agent detection to deliver a low-fi version of their mobile websites, allowing visitors using less capable smartphone devices to transact online instead of simply saying “your device isn’t supported”.

Smartphone Bot Indexing

While this has not been stated outright, there has been no mention of a new smartphone index, and historical Google mobile rankings indicate that they think that it is perfectly acceptable to rank and serve desktop pages to smartphone searchers.

This is also in-line with the recent history of Google updates, in which Google appears to be trying to speed up the delivery of their SERPs by not having to query multiple indexes, but instead tagging all entries in one index with specific ‘Universal’ ranking information – exactly what happened with the launch of Caffeine in June of 2010. (Why the separate mobile WAP index still exists is up for debate, but will be covered in another blog post.)

As far as we know, the new smartphone bot is just attaching alternate mobile attributes and meta data to pages that exist in the desktop index.

In the case that smartphone-optimized pages exist without desktop counterparts, these pages are also getting added to the primary Google index, with mobile indicators, to hopefully prevent them from appearing in desktop search results (or at least, that is what has been implied by Google).

Al explained that the mobile-optimized urls on Mantra.com.au were not showing up in ALL searches, and this he supposed was related to the nature of the redirects and the quality of the pages. In his own words:

“What is interesting is that not all URLs that we have specific redirects for seem to show the optimized URL & I’m not quite sure what the driver for that is. I was thinking that maybe Google have a quality signal associated to whether or not they’ll optimize the URL or not. For example, we use the same breadcrumb markup throughout our site but not all sections of our site show breadcrumbs in the search results – which I assume is a quality/link signal not being strong enough for those particular sections of our sites & I was thinking maybe the same sort of thing exists for mobile optimized URLs”

It could also be that not all of the pages have been crawled and indexed by the new smartphone bot, or some other less obvious problem that the smartphone bot found with the redirect, the content or the similarity of the two pages. Illustrations of the differences in indexing are included below.

A Page Affected By The Smartphone Bot

Where the desktop page used to be, the mobile page now ranks in its place. The link goes directly to the mobile property page without needing to be redirected. The listing shows the Desktop Title Tag and Description, but the mobile page url.

The smartphone listing links directly to the mobile optimized page

 A Page Not Affected By The Smartphone Bot

The desktop page is requested from the search result, but the server redirects to the mobile home page because there is not mobile page that is a direct match to this desktop page.

The listing that shows the desktop site is redirected to the mobile home page

Smartphone Bot Ranking

The most interesting thing about the activity and rankings of the smartphone bot crawl is that the results are not universal across the site, and appear to be keyword dependent.

Generic iPhone and Android queries on the brand name or brand+product were still returning the desktop home page url. The category and top level pages did not yet have a corresponding mobile page, so it makes sense for them, but the homepage did have a corresponding mobile page, so this was a bit odd.

In Al’s words:

“While we’re obviously redirecting our desktop home page to the mobile home page, none of our primary brand terms produce an optimised URL for the home page. ….At the moment, I have specific redirects in place for the ‘property’ content and all other URLs fall back to the mobile home page. As such, you’ll get mobile optimized URLs showing for queries that return hotel content such as [mantra 2 bond st], [mantra crown towers], [mantra circle on cavill] but not a generic query like [mantra sydney] or [mantra melbourne] – unless we have a hotel page that ranks for that term as well.”

Also interesting is the changes in the mobile sitelinks, which are sometimes different between desktop and mobile queries.

Apparently, the new smartphone crawler is choosing its own sitelinks, presumably based on their perceived relevance or optimization for mobile users, and including those instead of the sitelinks that are returned in a desktop search, as shown below:

Mobile Sitelinks	Desktop Sitelinks
Site Links Differ from Mobile To Desktop

What I thought was interesting was that some of the sitelinks returned in the mobile SERP with the desktop homepage listed were to pages that did not have a mobilized version, (for instance Mantra Resorts). This also occured when a mobile url was included with sitelinks, the sitelinks did not always go to pages that had been mobilized, shown below.

Mobile Sitelinks

When in portrait mode, there was only one sitelink, but two are displayed in landscape:

One Site Link in Portrait but Two in Landscape, All to Non-Mobile Pages

Desktop Sitelinks

This time, the sitelinks are the same on mobile and desktop, except desktop has one more: 

Three Site Links for Desktop

This is totally baffling, because on one hand, it appears that the smartphone bot is changing which site links appear, but it is not doing it in a mobile-optimized way.

Here is what Al had to say about that:

“I see room for improvement from Google on this front for mobilising the breadcrumb URLs displayed in the SERPs as well. Currently the title links to the mobile URL but the breadcrumb URLs are the same as the desktop URLs, for which we don’t have mobile specific pages or redirects in place just yet. I’d be great in the future if the “Gold Coast” link in the screenshot also linked to a mobile optimized URL (when we release those URLs into the mobile sites). It is possible that could happen currently but because we don’t have those type of pages on the mobile site & therefore don’t have the redirects, the breadcrumb in the SERPs isn’t being optimized yet.”

To me, this seems like a very bad user experience, because searchers expect to be delivered to a mobilized version of the Gold Coast page (reinforced by the title, description and bread crumb trail that are all pulled in from the desktop ranking), but instead are delivered to the mobile home page, where they will not be able to find the Gold Coast page, because it has yet to be mobilized.

In this case, it would be better if Google had left the desktop page link, because at least that has the information that the searcher was looking for. Hopefully this is a glitch that will quickly be fixed!

Conclusion

So that is the data. It does seem like the smartphone bot is a bit quirky, and there are a lot of odd things that are going on. It is odd that the smartphone bot is affecting sites like Mantra.com.au before sites like Facebook and Twitter. Odd that it is not updating home page urls in mobile search results, and results more likely generated by long-tail searches than generics. It’s also odd that it is changing sitelinks but not to make them direct people to pages that are mobile friendly.

Perhaps the new smartphone bot is on vacation somewhere in Australia, relaxing before full-on, logical deployment, but we will have to wait and find out. If you have interesting experiences, or similar experiences, please post them in the comments – we would love to hear them!