5 Tips For Optimal Mobile Site Indexing

It’s probably a sign of the times, but lately I have been fielding a lot of questions regarding SEO for mobile websites. Because there are so many ways to build a mobile site and because there are such a variety of mobile devices, one area that really piques my interest is mobile site indexing.

There are several tips and varied opinions on mobile SEO out there, and until recently, neither Google nor Bing have been very clear about mobile indexing best practices, especially regarding mobile indexing and smartphones. My goal is to offer up some of the most current best practices and research from the world of mobile SEO.

Judging by some of the forums and discussions out there, this topic is anything but resolved, and there are some heated opinions. Please let me know in the comments if I have missed any techniques and/or tips that have worked for you.

Mobile Site Indexing

First, we need to discuss how mobile sites are served to mobile searchers. One of the most popular methods is to create a user-agent detection at the server level.

If the user-agent is a desktop computer, serve the desktop version of your site. If the user-agent identifies itself as coming from a phone, a tablet, an iPad or anything that is mobile, serve that user the mobile version of your site.

I know what you’re thinking: Isn’t that cloaking? Nope. Don’t believe me? Just ask Matt Cutts. In this video, he’s pretty clear about it not being a cloaking issue (read: if you do it correctly).

The most common way of implementing user-agent detection is to place the mobile version of your website on a subdomain such as m.domain.com. Whenever a mobile searcher is directed to one of the pages from www.domain.com, a server-side 301 redirect will send them to the corresponding page at m.domain.com. (By the way, in the video I referenced earlier, Matt Cutts says that it’s a good idea to use this type of setup.)

Now, that doesn’t mean that you can’t do it other ways. Here is a list of other options I have seen companies use for mobile websites:

  • Subdomains:
    • m.domain.com
    • mobile.domain.com
    • touch.domain.com
    • wap.domain.com
  • Other TLDs:
    • domain.mobi
    • domainmobile.com
  • Directories
    • www.domain.com/m
    • www.domain.com/mobile

SearchEngineLand.com detects the user-agent and if it matches a mobile user-agent, the visitor will see a mobile-themed version of the site – but the URLs don’t change. The URLs are the same for desktop visitors and mobile visitors, and at the bottom of each page, the visitor can choose between viewing the regular version of the site or the mobile version.

This strategy is also great for indexing and PageRank, as there are no other unique URLs for the mobile version of the site. (Yes, I still pay attention to PageRank even though it makes a lot of SEOs LOL. Whatever. Don’t hold it against me.)

The only thing you need to be clear on is which method is best for you, in terms of technical feasibility and even in terms of analytics and tracking. There is no *right* way to do it, and there are pros and cons for each method. Pick the one that works best for your company’s situation, and don’t look back.

The next question is: How do you let Google and Bing know about the mobile version of your site?

Here are a few key tips:

1. Verify Your Mobile Site In Google & Bing Webmaster Tools

This works for mobile sites on their own TLD or subdomain, and it’s very helpful for getting data specifically about the mobile site.

2. Build & Submit A Sitemap Of Your Mobile Website’s URLs To Google & Bing

Google has specific criteria for mobile sitemaps:

	<?xml version="1.0" encoding="UTF-8" ?>
	<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0">	

The only real distinction here is the addition of the <mobile:mobile/> tag. Otherwise it is very similar to a typical XML sitemap. For more information on mobile sitemaps, please check out this post from Google’s WebmasterCentral blog.

3. Mobile Searchbots & Robots.txt File Considerations

Some webmasters and SEOs are noticing their desktop sites URLs in the mobile SERPs and their mobile URLs in the desktop SERPs. It is not happening all the time, and on the mobile side of things, it’s pretty standard nowadays.

However, for SEOs who want complete control over user-experience and/or branding, these blended search results are a headache, especially if they don’t have user-agent detection and a redirect configured.

Some of these SEOs go as far as blocking Googlebot from their mobile site and blocking Googlebot-Mobile from their desktop site. Here’s how that looks in the robots.txt files:

  • Desktop site: http://www.domain.com/robots.txt

    User-agent: Googlebot
    User-agent: Slurp
    User-agent: bingbot
    Allow: /
    User-agent: Googlebot-Mobile
    User-Agent: YahooSeeker/M1A1-R2D2
    User-Agent: MSNBOT_Mobile
    Disallow: /
  • Mobile site: http://m.domain.com/robots.txt

    User-agent: Googlebot
    User-agent: Slurp
    User-agent: bingbot
    Disallow: /
    User-agent: Googlebot-Mobile
    User-Agent: YahooSeeker/M1A1-R2D2
    User-Agent: MSNBOT_Mobile
    Allow: /

First off, how many of you knew that each search engine has its own mobile bot? Pretty cool, right?

To be clear, I do not recommend using this strategy of restricting bots. It is important that Googlebot has access to your mobile site and that Googlebot-Mobile has access to your main site.

My main reasoning: it’s probably not a good idea to block Googlebot from anything that you want indexed, even if it is a mobile site and even if the mobile SERPs are determined by Googlebot-Mobile. And it’s definitely not a good idea to block Googlebot-Mobile from your main site, as desktop URLs are still showing up a lot for searches on mobile devices.

Furthermore, trying to tell Google how to do their job (i.e. which results to show for any given search in any given setting) is a little arrogant, and ultimately it’s a waste of time.

It reminds me of back in the day when people used a meta tag to try to tell Google when to re-visit and/or re-index a URL. Yeah. Good luck with that.

Also, if you block Googlebot, get ready for an ‘unhealthy site status’ message in your Google Webmaster Central profile.

Finally, I’d like to point out two great quotes on this topic:

  • Googlebot, our crawler, must crawl your site before it can be included in our search index. – Jun Mukai, Software Engineer, Mobile Search Team at Google (source)
  • Don’t block the mobile site from being returned in desktop or smartphone results, as it’s relevant to more than just the search engine’s mobile index. – Bryson Meunier (source)

Recent News Regarding Mobile Searchbots

Google recently announced additional smartphone user-agent strings for Googlebot-Mobile:

  • Feature phones Googlebot-Mobile:

    • SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/ (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
    • DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  • Smartphone Googlebot-Mobile:

    • Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

Well, well, well. It appears Google is aiming to improve user experience on mobile search. I am very happy to see them doing something like this because there are a lot of mobile search marketers out there with specific needs and questions regarding mobile SEO, and some of the more specific issues are still a little cloudy. I don’t think this is the end of the discussion. In fact, in many ways, it is just the beginning.

4. Link To Your Mobile Site From Your Main Site & Vice Versa

For my penultimate mobile site indexing tip, I’m going to go old school on you. If you want URLs from a subdomain indexed, link to them from the main site. Linking between your main site and your mobile site will help with the crawling and indexing of the mobile site.

5. Begin A Link Building Campaign…But For Your Mobile Site

Because Google dominates market share for mobile search and because Google makes links such an important factor in their search algorithms, it’s a good idea to get external links to your mobile site. I’d start with mobile directories. Bryson Meunier has a list of mobile directories that will help get you started.

In your mobile SEO journey, be sure to check to see if your mobile site is being indexed and/or showing up in Google Mobile Search. I like to check to see what types of results are showing up for various search keywords and phrases at http://www.google.com/m/.

It’s a good idea to get familiar with the layout of mobile SERPs, and it will give you the chance to see how Google displays your site.

Well, that is mobile site indexing in a nutshell. Please share your thoughts and questions in the comments section below. I’m looking forward to the ongoing discussion.

Article image from Shutterstock, used under license.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: All Things SEO Column | Channel: SEO


About The Author: is Chief Traffic Officer at PMG, a full-service digital agency, where he directs all technical operations for the company, including the company’s SEO, social media, local search, reputation management, website architecture and usability, analytics and Wordpress practices. Follow him on Twitter @kerrydean.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • jhuman

    6. add interesting mobile snippets/excerpts/images w/tags

    Could you recommend a mobile ad network besides admob?

    I was wondering if you know of any hosting companies specializing in mobile?

  • http://www.raisemyrank.com/ Bob Gladstein

    Assuming you’re using user-agent detection to redirect mobile users to the mobile version of your site, and assuming you provide the option of mobile users switching back to the desktop version of the site, what would you say is the best strategy for keeping your server from redirecting those users right back to the mobile site? We’ve been considering cookies, checking the referrer, or linking to the desktop site with some parameter in the URL.

  • http://fr.linkedin.com/in/zinebaitbahajji zaitbahajji

    Thanks for all the explanations.
    I’m still a beginner in mobile SEO, and I have a question regarding robotx.txt files.
    We have a mobile version of our site, which does not look excatly like the website.
    It’s available on smartphones. and when a user access the website on a smartphone we display a message allowing him to switch to the mobile version.

    How would you write the robots.txt file ?
    would it be :
    user agent : *
    allow : /

    or should I list mobile bots ?

    Thanks in advance for your help.



Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide