How To Walk A Mile In A Search Engine’s Shoes

Small business owners are often curious, and sometimes desperate, to understand why their web sites are doing well — or doing poorly — when it comes to search engine visibility. Online forums and message boards are filled with questions like “Why is my competitor outranking me?”, “Why doesn’t my new product page bring me any search traffic?”, or “How come my site hasn’t been crawled in a month?”

If you live and breathe search marketing, these questions are often pretty easy to answer. But when you’re busy running a small business, these questions may as well be rocket science. One way to get answers is to analyze what the search engines think of your web site, and walk a mile in the search engines’ shoes, as the saying goes. When you learn to do that, it’s easier to solve those questions that you’ve been curious (or desperate) to answer.

Three ways to see what search engines think of your site

1. Use the search engines’ webmaster tools.

All three search engines offer tools for webmasters to help you understand how the engines see your site. At minimum, you should connect your site with Google’s Webmaster Central and Yahoo’s Site Explorer. MSN also recently launched its own Webmaster Center, and you should consider connecting your site there, too.

2. Use the search command to diagnose problems.

Even if you don’t use the various webmaster tools provided by the search engines, you can still learn a lot about what they think of your site with this command. Here are some:

  • How much of my site is in the index? If you have a 500-page web site, but the site: command shows that only 75 pages are indexed, you’ve just learned that you have crawling issues. It could be that your internal navigation is preventing the bots from reaching all your pages, or it could be that your site doesn’t have enough trust/authority to be crawled more deeply. (Getting more quality inbound links, especially to deep pages on your site, will help with the trust/authority problem.)
  • How often am I being crawled? Each search engine shows a “Cached” link with each result in the SERPs. If you click that link, you’ll learn when your pages were most recently crawled. What does it tell you? Crawl frequency is another sign of trust. If you’re not getting crawled regularly, work on creating quality content and attracting quality links. Check your main competitor regularly to see how often they’re being crawled — if it’s a lot more often than your site, set your goals accordingly.
  • Do I have duplicate content issues? I once worked with a client whose CMS published multiple copies of each page at different URLs. The home page, for example, was available at and The CMS also had e-commerce forms on a secure server, and it recreated the entire site on the secure server, so that had the same content as — and both were in the index. I quickly discovered all of these problems by using the site: command.
  • Are my page titles and meta description tags unique? This is another aspect of duplicate content. If you have a lot of pages with similar page titles and meta description tags, you’ll find fewer pages getting crawled and indexed. When you run the, it shouldn’t look like the image below.
Duplicate page titles and description tags

3. Use third-party SEO tools.

You can diagnose problems with third-party SEO tools that will help you learn more about how search engines view your site. Here are a few worth trying:

  • Index Rank will show you the rate at which Google is indexing your site. Use it to find indexing patterns, like how many new pages were indexed in the last month, three months, six months, etc.
  • SEO Browser will let you see your site the way a search engine spider sees it. Use it in conjunction with the idea above; if a search reveals that only 25% of your pages are in the index, SEO Browser might help identify crawling issues that spiders are having.
  • SEO Digger will show you a list of terms for which your site ranks in Google’s (or MSN’s) Top 20. The list this tool provides isn’t perfect, but it can be educational to see what terms you rank for, i.e. – what terms the search engine has decided to associate with your site.

Perhaps best of all, all three of these tools are free.

Final thoughts

Many small business owners don’t have the time or desire to become an SEO expert, but they still have important questions that need to be answered for their company’s long-term success. Connecting their sites with the official webmaster tools offered by each of the big three search engines is a great way to begin getting answers. Using the command on any search engine can provide additional information, and some third-party SEO tools can also reveal what a search engine thinks of their web site. Together, these tactics can help any small business owner walk a mile in the search engines’ shoes and solve those important questions that need to be answered.

Matt McGee offers search marketing consulting and training to businesses of all sizes. He blogs at Small Business SEM and The Small Is Beautiful column appears on Thursdays at Search Engine Land.

Related Topics: Channel: SEO | How To: SEO | Small Is Beautiful


About The Author: is Editor-In-Chief of Search Engine Land. His news career includes time spent in TV, radio, and print journalism. His web career continues to include a small number of SEO and social media consulting clients, as well as regular speaking engagements at marketing events around the U.S. He recently launched a site dedicated to Google Glass called Glass Almanac and also blogs at Small Business Search Marketing. Matt can be found on Twitter at @MattMcGee and/or on Google Plus. You can read Matt's disclosures on his personal blog.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.


Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide