Having A Crawlable Site Architecture Still Matters

Over the past few years, all of the search engines have made tremendous strides in trying to discover different parts of the web, find deep content and understand complex URL structures better. The first big stride came when each of the search engines adopted the XML Sitemap protocol. Website owners now had the ability to give the search engines a list of all of their URL’s.

A few weeks ago, another major step occurred when the search engines announced the adoption of a canonical URL tag to help them understand what the true URL is. The problem with this is that designers and developers are starting to use these tools as a crutch to compensate for a site architecture that is un-crawlable, poorly designed and unnecessarily confusing.

While you can use a sitemap to provide search engines with the URL’s, if you don’t provide a crawlable navigation scheme, you remove the power of internal anchor text from your website. Without that internal anchor text, you make it harder for the search engines to understand what your pages are about. Additionally, by removing the internal navigation, you lose the ability to help the search engines understand which parts of your website are important and set up a hierarchy.

Sitewide links across the top, side and footer areas help search engine find the high level key areas of your website. While there is an importance level tag for each URL in an XML sitemap, exactly how much weight, if any the search engines put on it, remains open for debate. Internal anchor text is big part of the puzzle, and removing it is like forgetting to put the sugar in your cookie recipe and expecting everything to come out of the oven tasting the same.

The new canonical tag solves the problem of pages with identical or nearly identical content existing for different URL’s. The search engines have indicated they will use this as signal to determine the proper URL but it is not an absolute declaration of the proper URL. For example, let’s assume this is a URL:


But people have the ability to sort page elements, producing URL’s like:




What happens when those URL’s start acquiring more backlinks than the pure URL, which is specified in the canonical tag – which URL will the engines prefer? Don’t think it matters? What if instead of a “sort” parameter you use an affiliate ID parameter, and it acquires the most backlinks, and starts to outrank the proper URL specified in the canonical tag? It might look something like this:


As a website owner, you might be giving away a commission to someone who didn’t deserve it.

The takeaway here is that you should design your website structure and CMS to use and be naturally crawlable, with search engine friendly URL structures. Tools like sitemaps and canonical tags should supplement, but never replace or displace standard crawling as the way your website interacts with search engines.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: All Things SEO Column | Channel: SEO


About The Author: has over 10 years experience in website development and internet marketing. He has helped companies with internal search engine optimization strategies for both ecommerce and informational websites. He publishes controversial industry thoughts and observations on his blog at www.wolf-howl.com

Connect with the author via: Email


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.


Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide