How To Walk A Mile In A Search Engine’s Shoes

Small business owners are often curious, and sometimes desperate, to understand why their web sites are doing well — or doing poorly — when it comes to search engine visibility. Online forums and message boards are filled with questions like “Why is my competitor outranking me?”, “Why doesn’t my new product page bring me any search traffic?”, or “How come my site hasn’t been crawled in a month?”

If you live and breathe search marketing, these questions are often pretty easy to answer. But when you’re busy running a small business, these questions may as well be rocket science. One way to get answers is to analyze what the search engines think of your web site, and walk a mile in the search engines’ shoes, as the saying goes. When you learn to do that, it’s easier to solve those questions that you’ve been curious (or desperate) to answer.

Three ways to see what search engines think of your site

1. Use the search engines’ webmaster tools.

All three search engines offer tools for webmasters to help you understand how the engines see your site. At minimum, you should connect your site with Google’s Webmaster Central and Yahoo’s Site Explorer. MSN also recently launched its own Webmaster Center, and you should consider connecting your site there, too.

2. Use the site:yourdomain.com search command to diagnose problems.

Even if you don’t use the various webmaster tools provided by the search engines, you can still learn a lot about what they think of your site with this command. Here are some:

• How much of my site is in the index? If you have a 500-page web site, but the site: command shows that only 75 pages are indexed, you’ve just learned that you have crawling issues. It could be that your internal navigation is preventing the bots from reaching all your pages, or it could be that your site doesn’t have enough trust/authority to be crawled more deeply. (Getting more quality inbound links, especially to deep pages on your site, will help with the trust/authority problem.)
• How often am I being crawled? Each search engine shows a “Cached” link with each result in the SERPs. If you click that link, you’ll learn when your pages were most recently crawled. What does it tell you? Crawl frequency is another sign of trust. If you’re not getting crawled regularly, work on creating quality content and attracting quality links. Check your main competitor regularly to see how often they’re being crawled — if it’s a lot more often than your site, set your goals accordingly.
• Do I have duplicate content issues? I once worked with a client whose CMS published multiple copies of each page at different URLs. The home page, for example, was available at domain.com and domain.com/home/. The CMS also had e-commerce forms on a secure server, and it recreated the entire site on the secure server, so that https://domain.com/about/ had the same content as http://domain.com/about/ — and both were in the index. I quickly discovered all of these problems by using the site: command.
• Are my page titles and meta description tags unique? This is another aspect of duplicate content. If you have a lot of pages with similar page titles and meta description tags, you’ll find fewer pages getting crawled and indexed. When you run the site:domain.com, it shouldn’t look like the image below.

3. Use third-party SEO tools.

You can diagnose problems with third-party SEO tools that will help you learn more about how search engines view your site. Here are a few worth trying:

• Index Rank will show you the rate at which Google is indexing your site. Use it to find indexing patterns, like how many new pages were indexed in the last month, three months, six months, etc.
• SEO Browser will let you see your site the way a search engine spider sees it. Use it in conjunction with the site:domain.com idea above; if a site:domain.com search reveals that only 25% of your pages are in the index, SEO Browser might help identify crawling issues that spiders are having.
• SEO Digger will show you a list of terms for which your site ranks in Google’s (or MSN’s) Top 20. The list this tool provides isn’t perfect, but it can be educational to see what terms you rank for, i.e. - what terms the search engine has decided to associate with your site.

Perhaps best of all, all three of these tools are free.

Final thoughts

Many small business owners don’t have the time or desire to become an SEO expert, but they still have important questions that need to be answered for their company’s long-term success. Connecting their sites with the official webmaster tools offered by each of the big three search engines is a great way to begin getting answers. Using the site:domain.com command on any search engine can provide additional information, and some third-party SEO tools can also reveal what a search engine thinks of their web site. Together, these tactics can help any small business owner walk a mile in the search engines’ shoes and solve those important questions that need to be answered.

Matt McGee offers search marketing consulting and training to businesses of all sizes. He blogs at Small Business SEM and HyperlocalBlogger.com. The Small Is Beautiful column appears on Thursdays at Search Engine Land.