SEO is an art. (Hence, the name of my and my co-authors’ brand new book, The Art of SEO). Crafting copy that sells, as well as ranks, is an art. So is link baiting. But SEO is also a science. Crafting rewrite rules, robots.txt directives, and so forth is pretty geeky stuff. The science side of SEO is where I spend most of my time.

Another dichotomy is that SEO is both subjective and objective. The point at which a title tag, URL, or headline is “good enough” and thus moving on to the next task is warranted — that is certainly subjective. Also consider what might comprise the most optimal URL structure? Does it end in / (slash) or a file extension like .html? Again, subjective.

In my view, SEO for the most part is cut-and-dry, it’s objective. That’s because it can all be boiled down to an algorithm, and in fact, it already has. The algorithm I speak of, of course, is Google’s (or Yahoo’s, or Bing’s). The SEO practitioner’s challenge is to reverse-engineer that algorithm to the best of their ability. But it shouldn’t stop there. Why not write your own algorithm — an approximation of the search engine’s own algorithm, one that teases out the various signals and accurately assesses the quality, relevance and importance of these signals without human intervention/assistance?

Running algorithmic analysis on a site-by-site and a page-by-page basis will then allow you to ascertain a site’s SEO health, and more importantly, the subsequent actions required in this never-ending process known as optimization. That is data-driven decision-making, my friends, and it will be a key driver in the next stage in the evolution of SEO.

To be effective, SEO scoring has to get granular. Knowing you scored an 89 out of 100, or a B+, overall with your SEO may be reassuring, but there weren’t any next steps that followed from that knowledge. The same is true even if you individually score each of the major SEO areas of focus. In my SEO Report Card column for Practical Ecommerce, I (arbitrarily) chose the following areas of focus: Home Page Content, Inbound Links and PageRank, Indexation, Internal, Hierarchical Linking Structure, HTML Templates and CSS, Secondary Page Content, Keyword Choices, Title Tags, and URLs. I don’t claim that these are the best “buckets”. Nonetheless, scoring such broad areas is still not actionable, really.

Score the title tags, internal anchor text, keyword prominence, H1s, meta descriptions and so forth separately, and on a page-by-page basis, and now you’re talking!

SEO effectiveness can be deconstructed into its many components. It can be benchmarked against competitors. Inferences can be made, priorities can be set, content can be massaged, link juice can be directed. Consequently, the SEO practitioner relies less on their gut and more on the data to drive their actions.

One enterprise-level SEO scoring technology that supports such a data-driven approach to SEO is Covario’s Organic Search Insight — which gets so granular that components of navigation, URLs and so forth are assessed (as can be seen in the screenshot below). Craig MacDonald, VP of Marketing and Product Management told me “The impact analysis can also be statistically modeled, based on gathering data across many sites over time in order to ascertain the relationship between changes in factors and the impact of those relative factors on the various search engines – i.e., the science can be rigorously applied.” From that, specific recommendations are automatically made and prioritized.

(click to view full size)

SEO is a moving target, one that is heavily dependent on algorithm shifts, site changes/updates, the competitive landscape in which one operates, etc. As such, you must continuously monitor and evaluate, ideally with an automated tool. In fact, such a tool is mandatory if you have a large site and you want your SEO activities to be scalable. With this monitoring in place, a page element (like a meta description) that goes AWOL can be flagged and the issue addressed (e.g. internal resources deployed) much faster than would be otherwise possible. Even better if the webmaster can be flashed warnings prior to making site modifications that will be detrimental to SEO.

The holiday shopping season is coming up fast. Don’t worry, there’s still time to do some quick-turnaround SEO that can have an impact on your natural search traffic (and resulting revenue!) in time for Black Friday and CyberMonday.

For many online retailers, November and December are the busiest months of the year. Of course, this is one of the most nerve-racking times of the year as well, especially when a disproportionate amount of one’s business hinges on such a short time span.

As you gear up for the holiday season, search marketing underpins your overall online marketing. Don’t neglect “search”, invest in it. Building on these holiday SEM tips from Niraj Shah, here are a few more SEO-specific tips to help you on your merry way…

Escape the code freeze

A common IT practice among mid to large size online merchants is to institute a “code freeze,” or site lockdown, during the holiday shopping season to minimize the potential for catastrophic errors and downtime. If your organization institutes a code freeze, you may not be able to make changes to your site for months. This means your on-page SEO effectively stagnates for an entire quarter. It also means you must race against the clock to implement SEO initiatives of any significance; and if you don’t make it in time, you must wait until the new year.

One way around this stress-inducing time crunch is to implement an SEO proxy platform. Such a system allows you to implement optimizations via the proxy throughout the holidays, quickly and easily, without impacting your native site.

Perhaps your code freeze still allows you to add static landing pages during the holiday season. That’s better than nothing, but it can take several weeks for new pages to make it into the search engines — and during the holidays, time is of the essence. So, the sooner you can add links to these new pages, the better.

Audit time

If you haven’t already, now is the time to do a quick audit of your site. If a code freeze is or will soon be in place, you won’t be able to make sweeping changes, but hopefully there’s still the opportunity to fix anything that isn’t working right. Especially if you just underwent any major changes in the last few months, this is a critical time to find anything that slipped through the cracks.

If you aren’t under a regimented code freeze, it may not be a bad idea to implement a self-imposed one as this can be a dangerous time to make major changes to your site.

Double check your contact form, live chat or other mechanisms that are in place for customer contact.

Review the last couple months of log files or your site analytics, looking for any 404 errors for missing files, moved or removed pages, broken links on your site, or missing graphics.

Make note of your most active pages. While you probably won’t be making any major changes to these pages, they might be good targets for including links to other key site pages to route holiday traffic.

Reorganize your internal links

Your customers probably have different buying habits during the holiday season than they do at any other time of the year. Therefore, it’s common sense that you should modify your internal linking structure to reflect seasonality. Don’t trash your existing site’s navigation, simply augment it with additional links containing keyword-rich anchor text, to create shortcuts that pass PageRank to your popular holiday categories and products. For example, if all of your holiday ornaments are three clicks away from your home page, create a text link on your home page that reads “Christmas ornaments” or “holiday ornaments.” Don’t rely on links on your site map page or on footer links to achieve this; such links are less than ideal.

Since hundreds of thousands of people search for phrases that include “gifts,” you would do well to create a Gift Ideas page for your specific industry/market, then optimize it and place it one click away from your homepage to maximize its PageRank and give it the best opportunity to rank well.

Don’t go overboard in your internal linking. Keep in mind that Google advises that you to keep the number of links on a page to fewer than 100.

Merry meta descriptions

Are you promoting a holiday sale or specific items on your site? If so, don’t forget to polish your meta descriptions so that searchers will recognize the keywords they’re looking for. Last-minute holiday shoppers will be attracted to descriptions that speak to them, so remember to mention seasonal search phrases to encourage them to click through to your site.

Including calls-to-action and/or value propositions into these meta descriptions will help ensure these searchers react favorably and click on your listings.

Be sure to make a list of the pages you revised so you can change the meta descriptions back after your New Year’s Day sale.

Pareto principle of link building

The Pareto Principle, also known as the 80/20 rule, says that 80 percent of the value/effects come from 20 percent of the causes. Arguably that concept could be applied to link building: 80% of your link authority (PageRank) comes from 20% of your back links. So your job is to focus on building more of those “vital few” links that deliver the bulk of your link authority.

Now is NOT the time to start some long, drawn-out link building initiatives to build these high-value links. There just aren’t enough weeks left in the 4th Quarter to do proper planning and execution for a complex and involved campaign, such as a music video creation contest. Focus on the “quick wins” – things like socially-seeded link bait articles hosted on your site, or single links that by themselves will have a measurable impact, acquired from sites where you have a relationship or some other “in”.

Have you been meaning to submit a guest article to a respected online publication that would love to publish your “thought leadership” and agrees to link to you from the byline/bio? No time like the present for that! One link from a trusted high PageRank authoritative source like that could boost your rankings in just weeks.

Also, use your influence with business partners and bloggers you know who already link to you, and try to get them to revise the anchor text of their links to you when the anchor text is less than ideal (e.g. “click here” or “visit site”). Again, focus on your most valuable links.

“Free” is a strong attractor

With giveaways like “free gift wrap” and “free shipping” you’ll attract holiday shoppers because you’re providing them with real value. Take advantage of this fact by incorporating powerful messaging (e.g. “free gift wrap,” free shipping”) into the title tags, body copy, and meta descriptions which will filter into the snippets of your search listings.

Even though it may not be free, another way to “give” during the holiday season is to offer gift certificates for the last-minute shopper. Feature gift certificates prominently on your site and cross-sell them on your “gifts” and “gift ideas” pages to achieve maximum visibility. Start optimizing for gift certificate related search terms through featuring your gift certificates immediately; don’t wait until the holiday season kicks into full swing.

Blog to attract customers and links

Hopefully you already have a blog. If not, then you’ve already found your New Year’s resolution. (What a relief to have that out of the way, eh!)

It should go without saying: make sure that your blog isn’t just another sales pitch. Your blog should be about connecting and communicating with your readers. Share some holiday stories, maybe your favorite recipes, or offer helpful packing and shipping tips. Any product mentions should be done carefully and subtly and in moderation. Blog with conviction and/or humor and/or personality. Offer real value. Be transparent, authentic.

If you don’t have time to blog yourself or with internal resource, you could try recruiting passionate customers as blog authors and run a group blog.

Map out your blogging for the season just like you map out your sales and advertising calendar. Plan posts now and start working on them for publishing later so that you can keep on top of them during the busy holiday season. Submit pre-written, post-dated blog content into your blog platform (e.g. WordPress) so you can maintain an active publishing schedule – even if you’re pressed for time and blogging seems daunting (if not impossible). That way, when you are inspired and free to generate a flurry of blog posts, all of these posts won’t be clumped together.

Always keep writing. Make sure posts are published regularly and frequently so that you have very few “gaps”.

Participate in the blogosphere

If all you do to engage with bloggers is to blog yourself, you’re really missing the mark. You should be spending as much time commenting on the blogs of important bloggers in your industry/segment/market as you spend writing for your own blog. That will help get you on the radar screen of these influencers. You can also generate positive buzz in the blogosphere by sending free product samples or review copies to these bloggers with “no strings attached” — it’s a bad idea to try to buy bloggers off by sending them useless kitsch. Remember that bloggers can wreak havoc on reputations, so tread carefully.

Get social

It is not too late to get out there, create viral content, and build your network of friends. You’ll need to work fast though. Whether it’s on YouTube, Facebook, Delicious, Flickr, Digg, etc. It can be as easy as publishing a killer list of gift suggestions and asking a power user friend to “seed” it into social sites like StumbleUpon and Kirtsy.

And remember, the #2 search engine isn’t Yahoo, it’s YouTube. If you’ve been waiting for your film debut, here’s your chance: produce a light-hearted, or humorous, or helpful video and post it to YouTube. It could be just the thing for that extra boost. If you have products that require complicated assembly, some short instructional videos would almost certainly be well-received; who knows, maybe they could become unexpected holiday hits.

Connecting offline and online

If you live and breathe SEO, offline may be the furthest thing from your mind. Of course, there are other angles to the offline world. Hopefully you have the basic fundamentals covered, like making sure that your web address appears in all your mailings, advertisements, and anywhere else you may be marketing offline. If you have brick-and-mortar locations, it doesn’t hurt to remind people that you also have a website.

Don’t forget that many of the traditional offline entities now have online presences themselves. TV news, radio stations, and newspaper reporters are often looking for interesting holiday stories, from the “most unusual” gift ideas to how to entertain for the holidays. Come up with an interesting story idea and you may not only get valuable airtime or print mention, but a link from their site to yours.

Separate out, into “buckets,” those purchases that happened offline (e.g. phone orders) but that resulted from online marketing (i.e. were generated from natural search, from paid search, from print, etc.). You could even go more granular, beyond the referral source, and associate actual keywords (search terms) with these referral sources.

With some cleverness, creativity and a bit of “elbow grease”, this holiday season could be your most successful yet, recession or no recession. And there’s still time, if you act now.

Nowadays, a great many websites  are powered by a content management system (CMS) along with a back-end database. And for good reason. It’s too unwieldy to code HTML on a page-by-page basis, as you expand your content offerings to the thousands or tens of thousands of pages (and beyond). Content managements systems to the rescue! But there can be downsides too.

My biggest gripe with the content management systems of today is their lack of SEO features. And I’m not talking about meta keywords, which are a complete waste of time.

I’m patiently waiting for the day when a CMS-based site can rival static HTML sites in SEO. No bones about it, hand-coded sites offer complete, granular control over each page, and every single tag contained within. That’s real flexibility. Too bad they don’t scale. Therefore, the SEO practitioner is going to need a CMS that will at least be cooperative.

Which SEO features should you be shopping for in a CMS? Glad you asked. Here’s my wish list of features, broken down into critical, important, desirable and optional…

Critical CMS features

• URLs free of tracking parameters and session IDs — Sticking session or tracking information such as the user’s clickpath into the URL is deadly for SEO. It usually leads to incomplete indexation and duplicate content issues.
• H1 tags — No H1 tags on a given page is not desirable. Too many H1 tags on the page is not desirable. Low-value content (such as the publication date) marked up as an H1 is not desirable. The article title is typically the best content to have wrapped in an H1.
• Customizable URL structure — If the default URL structure of the CMS doesn’t suit your needs, you should be able to change it. For example, if you don’t want /archives/ in the URLs of all your archived articles, you should be able to remove it. Or if you want to reference the article name instead of the article’s database ID in the URL, you should be able to do it.
• 301 redirects to canonical URL — Duplicate content is the bane of the existence of many a dynamic website owner. Automatic handling of this by the CMS through the use of 301 redirects is a must.

Important CMS features

• Static-looking URLs — The most palatable URLs to spiders are the ones that look like they lead to static pages, i.e. no query strings.
• Keywords in URLs — Keywords in your URLs can help your rankings. It would be a shame to miss out on the opportunity this presents, if your CMS doesn’t support keyword-rich URLs (e.g. only article IDs in the URL).
• RSS feeds — RSS feeds are essential if you want to reach bloggers; email newsletters won’t cut it for the hip, Web 2.0 crowd. Hopefully this feature also comes integrated with Feedburner, for improved visibility on RSS feed consumption by your subscribers.
• Pings — This lets blog and feed search engines like Google Blog Search know you have published new content so they can come and grab your latest RSS feed.
• Tagging and tag clouds — This Web 2.0 feature is powerful for SEO, thanks in large part to the keyword-rich text links. This is your opportunity to rejig your internal linking structure and how you flow PageRank without having to completely gut your taxonomy/ontology.
• Individually customizable title tags and H1 tags — Each title tag should be decoupled from the post/article/product title. Same goes for H1 tags. That way anchor text can be varied from H1’s which can, in turn, be varied from the title tag. Thus, you can work in additional keywords (synonyms etc.) into the H1, and even more into the title tag — without spamming of course!
• Multi-level categorization structure — It’s awfully limiting to your site structure and internal hierarchical linking structure to have a CMS that doesn’t allow you to nest subcategories into categories, sub-subcategories into subcategories, and so on.
• Canonical tags — Although I don’t trust Google to always reliably obey this new tag, it is definitely worthwhile having it available as an option if the need arises (hopefully that need won’t arise if you have 301’s in all the right places).

Desirable CMS features

• Paraphrasable excerpts — Duplicate content issues are exacerbated on dynamic sites such as blogs when the same content is displayed on permalink pages, category pages, archives-by-date pages, tag pages, and the home page. Crafting unique content for the excerpt and having that content display on all locations except for the the permalink page will help strengthen your permalink page as unique content.
• Breadcrumb navigation — It reinforces the hierarchical nature of your internal linking structure using text links which are hopefully keyword-rich.
• Flexible rules for automatically generating title tags — If the title tag always has to start with your site name followed by a colon followed by your article title, you’re sunk — at least as far as your SEO is concerned. You should be able to revise the “recipes” used to generate the title tags across your site to make them more optimal for search.
• Page-specific meta descriptions — A cardinal sin of dynamic websites is using the same meta description across all the pages. This can be a contributor to duplicate content issues.
• Meta noindex for low-value pages — Even if you nofollow links to these pages, other people may still link to these and you run the risk of ranking those pages above some of your more valuable content.
• Keyword-rich intro copy on category-level pages and tag pages — Keyword-rich introductory copy helps set a stable keyword theme/focus for the page, rather than relying on the latest article, product, or blog post to be the most prominent text on the page.
• Granular control over nofollows on links — If your site allows the posting of user-generated content through “comments,” your site will be a spam-magnet if you don’t nofollow the links posted by commenters. Heck, you’ll probably be a spam magnet anyways, it’ll just be worse for you without the nofollows. Additionally, regardless of your stance on PageRank sculpting and its value for SEO, you should be able to selectively decide when and when not to pass PageRank to an internal page within your site.
• Customizable anchor text on navigational links — “Contact”, “About Us”, “Read More”, “Full Article” etc. all make for lousy anchor text — at least from an SEO standpoint. Hopefully your CMS allows you to improve such links to make the anchor text more keyword-rich.
• Mass Edit, or Bulk Upload (or both) — It’s not efficient to go to each page’s Edit screen. Instead, mass modify the titles, H1’s, filenames, and perhaps even meta descriptions, within Excel or a “mass edit” web interface (like the one provided by my SEO Title Tag plugin for WordPress.
• Declared search term — When you decide on a page’s primary keyword focus, you should be able to tuck away that crucial bit of information somewhere where it will be safe from the prying eyes of competitors. That means it should not be parked anywhere in the HTML — including the meta keywords tag — since all a resourceful competitor would need to do is “View Page Source” within their web browser. There should be a field in the database, displayed and accessible to your editors/administrators within the admin interface of your CMS.
• Auto 301 redirect previous versions of URLs — Imagine updating a permalink or product page URL (e.g. “post slug”) multiple times. Each previous version of a URL could lead the search engines to discover duplicate pages if you’re not careful. Why worry about these old URLs and whether they will stop working or will create duplicate content; let the CMS “worry” about this instead and seamlessly 301 previous iterations to the latest version.
• Google Product Search feed — If your CMS is powering an online catalog site, then this feature is for you. It can be a real timesaver. And if you are an online retailer not submitting your products into Google Base, heed this warning: neglect Google Product Search (formerly Froogle) at your peril!

Optional CMS features

• XML Sitemaps generator — A XML sitemap can be submitted to the major engines to improve indexation, but it’s usually unnecessary if you have a search engine friendly CMS; the engines will usually do a good job crawling and discovering your site’s URLs on their own. Google will use your Sitemaps file as a canonicalization signal, but hopefully you don’t need it since your CMS isn’t generating duplicate pages.
• XHTML validation — When entering your content, it is desirable to have the CMS automatically check for malformed HTML, as search engines may end up “seeing” a page differently from how it renders on the screen and consider navigation to be part of the content or vice versa.
• Pingbacks, Trackbacks, Comments and Anti-spam mechanisms — The problem with comments/trackbacks/pingbacks is that they are vectors for spam, so if you have one (comments/trackbacks/pingbacks), you will have the other (spam). Therefore, effective spam prevention (e.g. Akismet, Defensio, Mollom) is a must.

Links long ago became the currency of the Web, thanks in no small part to Google and its PageRank algorithm. Like anything of value, link authority is bought, sold, leased, bartered, brokered, swindled and stolen. The fact that links are valuable is widely accepted. But yet when pressed, can any SEO or link builder really say what that value equates to in real dollars? Of course, attributing sales revenue to direct click-through traffic (from the linking page) is a straightforward exercise (albeit sometimes an incomplete one, if/when the referring URL data is scrubbed). But what about the SEO impact of that particular link on the linked-to page? Or even harder to measure: the “rising tide that lifts all boats” effect that the link has across the site by boosting nearby pages / sitewide PageRank / domain authority? Is that value quantifiable?

Many times, we’re asked by prospects to substantiate our claim that link building (when done well) is a high ROI generating activity, and clients also ask us to track and report on the impact of our link building efforts. You can track and tally your link conquests easily enough with advanced link building tools like SQUID, Raven and BuzzStream. It’s quite another matter to assign a specific dollar value to the links — at least one with any accuracy.

For example, in my last column, I described the process for generating link bait and seeding it into social media that we practice at Netconcepts. Let’s say you are paying a retainer of $20k per month for this service (ideation, research, writing, seeding, competitive intelligence, phone & email reach-outs), and let’s say that hundreds, or perhaps thousands, of quality links per month result from the link builder’s activities. What’s the ROI on that $20k spend? Now, to complicate matters, let’s say the link builder works to acquire specific high-value links one-at-a-time, in a “brute force” fashion using hand-crafted emails and phone calls? Which links account for which lift in rankings? Even ascertaining which links were discounted due to spam signals is difficult to impossible, at least in any scalable way.

Consider this specific link building campaign. Netconcepts dreamed up and orchestrated a “Free Business Cards for Life” contest for our client Overnight Prints, that involved the self-made millionaire, Internet celebrity and Technorati Top 100 blogger Jeremy Schoemaker (aka “Shoemoney”). The contest was to design Jeremy’s business card. The winning entry, chosen by Jeremy out of over 400 submissions, was amazing. Our client, Overnight Prints, received some great links, with some great anchor text, from the initiative. Don’t ask me to quantify the value of these links though, because I can’t. But I just “know” that this campaign will have an impact.

It’s an inexact science. Organic rankings and search traffic across the client’s keyword portfolio, pre- and post-campaign (by at least 90 days), is a good starting point, but is by no means sufficient for the ROI-focused marketer. Both the revenue line and the cost line must be pinned down, somehow. Yet the link builder would likely counter with the argument that revenue is too many steps removed from what they actually have control or influence over, that growth in the quantity and quality of the backlinks is more appropriate in gauging success and justifying their salary — assuming that the fruit borne from their efforts can be isolated from other marketing activities (e.g. email marketing, direct mail), and from naturally occurring link growth.

All this makes selling a client or boss on the value of link building a tricky proposition. And it dissuades that decision-maker from allocating budget or authorizing the project. Nobody wants to simply “take your word for it” and whip out their checkbook.

Despite the aforementioned, the linking ecosystem continues to thrive. The suppliers to the ecosystem — from the expert individual practitioner (such as Eric Ward) to a reputable link broker (one of Conductor’s lines of business) — are inundated with work. Plenty of disreputable firms also feed at the trough — whether they be link spammers out of India polluting the blogosphere with their useless blog comments and filling webmasters’ inboxes with annoying link requests, or the link spam software providers that aid and abet them.

Given the attribution challenges, it can be tempting to scrimp on the vendor, consultant, or service hired for this all-important job. Remember though that you always get what you pay for. The wrong choice may ultimately do more harm than good. With no independent clearinghouse to vet link building services, one of the most important steps you can take is to do your homework on your potential vendor — reference checks, backlink analysis on them and their clients, and the like. Evaluating the business practices, business model, effectiveness, and business viability — this is all a must. Countless link building services have come and gone; you want to be sure you’re backing the right horse. Eric Ward advises that each link target requires an approach tailored to the unique nature of the target and the requester (features, content, audience, etc.). Eric warns: “cookie cutter packaged link building services will never put you in the best light.”

Furthermore, the “wing it” approach to link building is no longer viable, if it ever was. What’s required is the application of the scientific method in your link campaigns –- isolating control groups (both keywords and pages) that can be juxtaposed with the experimental group in order to gauge effectiveness of various tactics and campaigns. Competitive SEO now requires the examination of large amounts of backlink data from multiple competing sites to tease out what is meaningful in the inbound link graphs.

This takes a lot of brain power (and math), but this is where we’re heading with the next generation of auditing tools. SEO technology solutions (e.g. Conductor, Covario, Enquisite, GravityStream, and others) promise us unprecedented ability to track, monitor, and assess on-page and off-page elements for you and perhaps even your competitors, all in a single place. Automating the process of analyzing and quantifying the depth and quality of inbound links is essential to achieving any kind of scale.

Per my recent Pay-for-Performance SEO article, I make the case that all SEO activities (including link building) can be valued and ROI calculated in aggregate (such as through the Enquisite Campaign product). But let’s go back to the original premise for this article: what about on a link-by-link basis? I want to be able to say with some certainty that a particular link is worth $X and another link is worth $Y. Indeed, within Campaign (and probably other solutions too), links can have different values based on the keywords they are connected to. Each keyword has a different click (Campaign provides a market-based organic CPC for each keyword) and transaction value, and so does the link.

Technologies such as the aforementioned will herald in a new age where acquired links are tied to particular keywords, assigned a cost-per-click and/or conversion value, pegged as incremental over the baseline, attributed the referral traffic from direct clicks on the links as well as from the search traffic derived from the rankings lift, and so on. This I anticipate will do wonders for legitimizing link building to the direct marketers of the world.

A problem arises, however, when link builders are held responsible for missing ROI targets (regardless of how reasonable those ROI targets might be). This would be unfortunate, because link builders are not in control of the outcome — the link placement, URL target, anchor text, not to mention whether the link request is granted in the first place. In fact, the higher the credibility of the linking site (and thus the greater the trust, authority and importance in the eyes of the search engines), the less influence the link builder tends to have. Eric Ward describes such high caliber site owners as “passionately picky about what does and does not get on their pages.” And he would know; he’s built links for over 1000 sites over 15 years.

So by all means, predict, goal-set, track, analyze and cost-justify your link building initiatives. Eric just asks that you don’t hold your link builders accountable for editorial decisions made on sites they do not control.

Building links is a struggle we SEOs all face. Of the three pillars of SEO (content, architecture, and links), it’s the “link authority” pillar that i’s usually the weakest. Looking at sites individually, formulating your approach, sending personalized emails, picking up the phone to speak to the webmasters – it’s a lot of hard slog. If only it weren’t so darned difficult and time-consuming to acquire high quality, relevant links! Yet without such links, you won’t be able to earn the trust, authority and importance required to rank, and your optimization efforts will fall short.

There is another approach – a “secret formula” if you will, employed by the SEO elite. One that’s scalable, efficient and high-impact.

It starts with “link bait”. Yes, that overused, industry term that refers to viral content that is irresistible to link to. But link bait by itself isn’t sufficient. You need to “seed” this link bait into social media (such as Digg and StumbleUpon) using power user accounts within those communities. In other words, you need to be (or be able to call in a favor with) a social media insider who has wide-reaching influence within that site’s social community. Without a bevy of friends, followers and fans, it’s much harder to reach “escape velocity” quick enough. That’s because the algorithms for “what’s new and hot” within social news/bookmarking sites take into account the time span within which the positive votes are acquired. A thousand Diggs over a year is a very different thing from a thousand Diggs over 24 hours.

The “secret formula” really is a formula – or should I say, formulaic. Think of it as an assembly line process. Viral ideas are generated. The chosen ideas are researched and written up as articles (or produced as videos), then published to your website. The articles are then seeded into appropriate social sites by influentials within those sites’ communities. The point of all this isn’t to reach your target market directly; these users probably won’t buy anything from you. It’s to reach that small percentage of the stampede to your site that are journalists or bloggers – who will write about and link to your viral content.

Let’s take a look at this process in greater depth.

Ideation: It all starts with a great idea and this step is key: your content/angle must be more than just clever to go viral. Gather a team of your most creative and knowledgeable SEOs and marketers (and/or your outside SEO firm) to brainstorm possible ideas for link bait that will likely resonate within social media. Develop a list of ideas; you can start with a handful and save the others for future use.

Selection and prioritization: Once a solid list has been developed, it’s time to prioritize and select the top several (e.g. three or four) ideas to develop into full-blown articles/blog posts. Be forewarned: you will have to stretch outside of your comfort zone; the edgier (or geekier, or low-brow) the article, the more it will resonate with the 16-year-old alpha geeks on Digg.

Content creation: Flesh out the chosen ideas into full-blown articles/blog posts starting with the research. For example, a topic of “Top 100 Beers from Around the World” will likely require many pages of information to be collected — from each beer’s history, to photos of the bottle/label, to nutrition information.

Make sure to craft a killer headline using this formula from social media marketer Muhammad Saleem: number + adjective + key phrase. e.g. “13 Most Chilling Haunted Hotels” or “16 Incredibly Unconventional Hotel Rooms.” It’s a catchy title that will reel people in. You may also/instead wish to develop video or other visuals to help support your idea.

You really need a “hook” to turn an article idea into something that will have “legs” in the social sphere. For example, consider a contest idea of “win free business cards for life.” Pretty ho-hum, right?

At Netconcepts, we developed a contest for our client Overnightprints.com. What was the angle/spin we added to the free business cards for life idea? Simply this: design Shoemoney’s (Jeremy Schoemaker’s) business card, Jeremy will serve as the judge and the winning entry (as determined by Jeremy) will get “free business cards for life.”

The cost of the contest for our client was negligible – the fine print of the contest stated the winnings were a maximum of 1,000 business cards per year for up to 20 years. Jeremy blogged on his Technorati top 100 blog, shoemoney.com, about the contest; he even posted a video about it to YouTube. The link exposure this contest garnered was priceless. Contests prove you don’t have to have a big budget — just a big idea.

Website Prep: Just like you’d prep for surgery, you’d prep your site for a social push. Brent Csutoras describes the process required here as getting your site “social media ready.” For example, if your site is a blog, it would be advisable to distance your site from blogs. Blogs are old and tired and not popular with Digg users anymore. So switch from a bloggy theme to a magazine style theme, and remove the bloggy references, such as date-based archives links and permalinks verbiage.

Also, make sure your site can handle the anticipated traffic. If your server buckles under the load and the site goes dark, your submission doesn’t just get pulled from Digg’s home page, it gets removed from the site altogether -i.e. completely obliterated. With Digg, hitting the front page will generate a traffic spike that will quickly dissipate and then disappear almost completely after 24 to 48 hours.

Publishing: You will need a place to host your linkbait; it really should be on your site if at all possible. Using a third party website to host the article will result in subpar performance; so hopefully you can tolerate the article living on your website. Remember you don’t have to link to the article from your navigation or sitemap or from anywhere else on your site. Also, you need to be cognizant of the time and day you publish. Publishing on Saturday night would not be a good idea.

Social Media Seeding: Immediately after you have published the viral content, it then needs to be submitted to the appropriate social media site(s). These may include social news sites such as Digg or social bookmarking sites such as del.icio.us.

As already alluded to, in order to maximize the chance of success, it’s important that the submitter be a “power user” within these social networks to maximize the chances of hitting the front page or “popular” page, thus driving the most visibility, traffic, and -– eventually –- links. A Digg submission from the great unwashed just won’t get the same traction from the social media community. A power user’s street cred is worth a lot, so be ready to pay for it — in terms of cash or favors.

You can’t be a power user across all the pertinent social media — it requires simply too much work. Power users spend all day and all night at their computer monitoring various oddball news RSS feeds and other social sites for content that will resonate with their targeted social site’s users. Once they find something, they quickly submit the URL along with a killer title and description – before anyone else can. This is how users move up in the pecking order within these social media. The more stories they can get to the front page, the higher their status.

Nearly everything that a top-ranked Digg user touches turns to gold; it’s not atypical for 80+% of their submissions to hit the Digg.com front page. As you can see, the power user is the key to this equation. If you happen to know one or be one, then you are in luck. If not, you might want to consider hiring a top-notch SEO and social media agency. Also, have a look at the Top 100 list of Diggers for the most popular power users, note however, real names are rarely attached.

Make friends within these groups and call in favors as you need them. In return, be prepared to do the same. Greg Boser says for every hour you spend working on a client in social, you need to spend six hours doing favors in return for those in your elite network with whom you called in a favor. It’s all about reciprocating. The more you vote and submit for others, the more you can ask for in return.

Now let’s talk results; an ideal result is your link bait hitting the front page of Digg and accumulating a thousand-plus links over the following 3 to 6 months. Why does it take so long when a Digg spike is so immediate? Because bloggers don’t blog about your link bait right away. They may keep it in the hopper and not get to it for weeks or even months.

Then, because the blogosphere is one big echo chamber, that blogger’s readership will include other bloggers who will eventually blog about and link to the article too. What would be the yield of really “hitting it out of the park” in terms of this sort of link building? Potentially over 5,000 links with perhaps a hundred of those being PageRank 7 and a half dozen being PageRank 8 (hypothetically). That’s a solid investment.

Link baiting is a critical tactic to have in your SEO “toolkit” because the efforts of just one article hitting the home page of Digg can do wonders for your link authority and thus your Google rankings. Put the time and effort into developing your most linkworthy ideas and go out and negotiate with your power user friends. This combination will ensure your success.

Hope you got some value out of me “spilling the beans” on this secret formula for generating linkbait, acquiring links and measuring success. Many marketers think of social media as a plaything; it would rarely be the tool of choice to generate serious ROI. Sure, social media IS fun, but it’s wrong to think that it can’t pay the bills. Leverage these social media outlets properly, and your link opportunities are endless.

The currency of the SEO industry is, and has always been, “dollars for hours” — in other words, the traditional consulting model that is in use at accounting and law firms around the globe. Just think of what would be possible if it were to change to something more tied to the value being delivered!

SEOs aspire to “knock it out of the park” for their clients. But they can’t always do so because fixed-rate contracts constrain them, limiting the resources they can feasibly (profitably) apply to any one campaign or client. For clients, it’s common for their SEO budgets to be nowhere near that of their AdWords budgets. And how could you blame them? Compared to PPC, it’s so much harder to isolate the impact of previous SEO activities, or to prove in advance the business case for an upcoming SEO initiative.

However, this becomes a self-fulfilling prophesy; it’s amazing how difficult it is to “move the needle” when you have insufficient resources. If only SEO could work like paid search and other direct marketing channels, i.e. like a veritable cash machine where you put $1 in and it spits $10 out…! Keep putting one dollar bills in, keep getting ten dollar bills out. You’d want to do that all day; certainly the CFO would. Problem is, it’s difficult to demonstrate unambiguously and reliably such a cash machine scenario with SEO. Until that changes, SEO will continue to be underfunded.

How do you shift the opinions of those who hold the purse strings — from SEO being an unpredictable “black art” to it being an accountable marketing channel? Simple: a pay-for-performance pricing model similar to that of paid search. This can only be possible when there is clear, comprehensive and objective tracking and measurement of SEO efforts and results. What I’m talking about is “search analytics” (including predictive analytics). This is different from “web analytics,” which wasn’t built by SEOs, gives short shrift to SEO, and really only tells you what you’ve already been getting rather than what you could be getting. Industrial-strength search analytics isn’t something we, as an industry, have historically had at our disposal. Until we do, the SEO agencies won’t be able to truly capitalize on the immense value created for their clients.

At Netconcepts, we have started down this road of accountable marketing. The way we “cracked the code” of pay-for-performance was by developing a scalable, automated natural search technology solution (GravityStream) and pricing it on a cost-per-click basis. Nonetheless, we still have flat-fee pricing for our SEO audits and monthly retainers for our ongoing SEO consulting, just like everyone else. Whether you’re client side or agency side, the Holy Grail is a pricing model based on the value delivered. It’s a win-win for both sides: for the vendor there is upside potential when they over-deliver (although ceilings or caps on that upside potential quickly evaporate much of the incentive), for the client much of the risk has been removed (i.e. only pay for actual clicks or other actions).

Today, this model is fraught with problems. An SEO might be reluctant to enter into such a performance-based arrangement because the actual implementation of his or her recommendations falls to the client (and, as such, is outside their control). The SEO is at the mercy of the client and its IT Department, and the implementation (or lack thereof) directly impacts revenue opportunity. In this scenario, the vendor is betting his/her paycheck on whether IT “gets it” and considers these modifications mission-critical. That is a bet with very bad odds.

So as an SEO consultant, how could you possibly adopt a performance-based pricing model without a crystal ball? By employing the newly launched (predictive) search analytics platform Enquisite Campaign. Let’s see how…

With this application, the first step is to create “opportunities” to help you determine whether a given SEO campaign is worth going after. You can estimate the number of potential referrals available, the revenue impact of those referrals, the amount of work required, and the resulting ROI. All these variables may be customized to the unique situation of a given vendor or client.

The potential — in terms of referrals, dollar value, and ROI — is calculated by Enquisite Campaign for all the keywords in your defined campaign list. In addition – here’s the crystal ball part – the application exposes keywords not on your original list that are either currently driving traffic or not currently driving traffic but related. Importantly, this is done without having any data on conversion rate, bounce rate, time on site, or pages viewed — data points which you normally need to determine the potential for keywords currently driving traffic. The number of referrals are estimated for keywords that are off your radar, based on keyword popularity data from Google and Yahoo (via their APIs) and click-through rate (CTR). If there is traffic already coming in to your site for a keyword, those referrals are pulled out and shown separately in the reporting so that you can clearly see the upside potential versus baseline traffic.

Here is where the customized variables come into effect. Client revenue from new transactions (i.e. selling stuff) is estimated on that potential traffic by multiplying the incremental search referrals, the conversion rate (as defined by you in a configuration screen) and the average order value (AOV, also defined by you). A value for a non-monetary action, such as a white paper download, can also be defined, along with the revenue potential for you. For example, if you’re IBM, you might value a white paper download as worth $200 based on how likely a white paper downloader is likely to become a six or seven figure consulting contract, some months down the line. Once the campaign is live, Enquisite Campaign then tracks all the incremental referrals, actions, and transactional conversions based on the real traffic and sales data from your site.

Currently, pay-for-performance is also tricky because when you’re trying to estimate ROI on a particular keyword, you need have a reasonable sense of what the cost of optimizing will be in man-hours. How do you do this when you don’t know the particular page to be optimized and what needs to be done to optimize it? As a vendor, you somehow need to know the “cost” (effort required) to acquire the associated revenue. The answer lies in estimating the “optimization difficulty” for each keyword.

Turns out there is a solid correlation between paid search competitiveness and organic optimization difficulty. So, by taking paid search data points, like the number of pages competing for the term, the number of search queries daily on Google and Yahoo, the number of bidders on paid search, the bids, and the CTR, Enquisite Campaign comes up with a reasonable approximation for optimization difficulty. Further, it’s completely independent of the tactics you’ll be employing or your particular circumstances — whether your challenges lie with the CMS, the IT department, or whatever.

Once the SEO nominates a budget in hours for the campaign, Enquisite’s application then allocates the hours across the identified keywords. If for example, you specify that you want to perform a maximum of 20 hours of SEO work in a month, Enquisite will divvy those 20 hours up across the keywords, assigning more time to the keywords that have the greatest upside for the least amount of effort. Now the ROI for each keyword can be calculated. In Campaign, the ROI is displayed as a percentage and calculated based on: revenue potential divided by the effective hourly rate times the estimated number of hours.

click to view full size

Once you know the ROI potential for any campaign, you can decide on what is the most appropriate performance-based pricing model to pitch to the client — whether it’s search referrals, sales, actions, or even website referrals.

Armed with the above intelligence, SEO practitioners are able to make a case internally for performance-based pricing. And by removing risk for the prospect, the barriers to saying “no” are removed. In a competitive bid situation, a performance-based pitch (presumably informed by data and analysis from Enquisite Campaign or similar) will make you stand out from the crowd because you have some serious “skin in the game.” Essentially, vendor and client incentives are aligned. If you drive incremental action, you participate in the value that is created. The client only pays for incremental upside, all clearly linked to objective data and results.

If you’ve read the book Freakonomics — one of my favorite business books of all time — you’ll know the extreme importance of such alignment. As with any competitive advantage, eventually others in the industry will take notice and decide to adopt (copy) the model. “Embrace and extend,” as Microsoft likes to say!

So there’s a window of opportunity here not to miss. First movers will be the biggest winners, as they will establish themselves as the true performers who are willing to bet on themselves.

What follows is a rant, which is something I rarely, if ever, do. It’s done in the spirit of fun, so don’t take it too seriously. Enjoy!

I feel like the grandpa who laments in a crotchety voice to his grandkids: “Nobody ever writes letters anymore! They just sit on their computers and their cell phones all damn day!” But instead I’m saying: “Nobody ever blogs anymore! They just tweet and re-tweet!”. For example, this tweet by @dannysullivan could have been a fantastic blog post. Instead: it’s 129 characters that merely hints at the story:

@dannysullivan: seriously, pinkberry with locations in 2 of 50 states ranks 14 for yogurt? http://bit.ly/dHOYe well @mattcutts does love them :)

Last week I had the pleasure of sharing the stage with Danny on a panel at the eMetrics Summit. The topic, unsurprisingly, was SEO, but targeted to web analytics geeks (a number of whom were SEO newbies). Danny kicked off the session with a quick SEO 101 where he expanded on his gem of a tweet above about Pinkberry.com. Pinkberry is a frozen yogurt brand that I was unaware of until the session. And what a brilliant example it was. Pinkberry.com is a case study in how NOT to build a website. I think they hired the Anti-SEO to ensure they wouldn’t rank for anything other than their brand name.

There was really silly stuff going on. Basic, basic on-page SEO was completely mucked up. Like for example, the page titles. Danny showed the audience site: results in Google for Pinkberry.com and the results were, well, disturbing to say the least—at least for anyone with an SEO bone in his/her body! Sure enough, every title tag was the same across the site. But wait, it gets better! The titles were all one word long: “Pinkberry®”. Luckily, the major engines don’t trip up on circle R and TM symbols, even when they are ASCII characters, or I’d be complaining about that too! (Nonetheless, I dislike such symbols in title tags. If you must use them in titles or elsewhere or you get yelled at by your legal department, then please “escape” them, e.g. ®—it’s just good HTML etiquette.)

Let’s move on to what is on the home page, that most important of pages from an SEO perspective. It’s a circa late 90’s “splash page”. With, you guessed it, zero textual content. This is what the home page looks like from a spider’s perspective. Pretty sad. Well, to be more technically correct, this is what it sees: there’s a single image with no alt attribute and a filename that is of no help whatsoever.

Moving on past the content-less splash page, you end up on a page where the mouseover navigation relies on JavaScript, which of course the spiders don’t support. Not only were the mouseover nav items inaccessible, but the main buttons (the ones available without hovering) stopped working. At least the ones that had mouseover effects attached to them. This included their “Products”, “About”, “Contact”, and “Groupie Corner”. Oh, and again, no textual content to be found. But hey, at least they had defined some meta keywords, so clearly someone at Pinkberry is at the wheel driving their SEO “strategery” (*grin*).

I think the only thing the Anti-SEO didn’t do was take any textual navigation or content elements that may have been remaining in spider-accessible formats/locations and wrapped a Flash movie around all of them. And perhaps added frames for good measure, complete with hidden links in the frameset pointing back to His site.

Yet somehow, despite themselves (as Danny notes in the tweet above) Pinkberry ranks on page 2 in Google for “yogurt!” Huh? Or as the younger generation like to say: “WTF??”

Matt Cutts, care to comment? Is this a result of your hand editing since you’re such a raving fan? Or, put another way (by the more politically incorrect SEOs out there like DaveN), a “hand job”?

The Robots Exclusion Protocol (REP) is not exactly a complicated protocol and its uses are fairly limited, and thus it’s usually given short shrift by SEOs. Yet there’s a lot more to it than you might think. Robots.txt has been with us for over 14 years, but how many of us knew that in addition to the disallow directive there’s a noindex directive that Googlebot obeys? That noindexed pages don’t end up in the index but disallowed pages do, and the latter can show up in the search results (albeit with less information since the spiders can’t see the page content)? That disallowed pages still accumulate PageRank? That robots.txt can accept a limited form of pattern matching? That, because of that last feature, you can selectively disallow not just directories but also particular filetypes (well, file extensions to be more exact)? That a robots.txt disallowed page can’t be accessed by the spiders, so they can’t read and obey a meta robots tag contained within the page?

A robots.txt file provides critical information for search engine spiders that crawl the web. Before these bots (does anyone say the full word “robots” anymore?) access pages of a site, they check to see if a robots.txt file exists. Doing so makes crawling the web more efficient, because the robots.txt file keeps the bots from accessing certain pages that should not be indexed by the search engines.

Having a robots.txt file is a best practice. Even just for the simple reason that some metrics programs will interpret the 404 response to the request for a missing robots.txt file as an error, which could result in erroneous performance reporting. But what goes in that robots.txt file? That’s the crux of it.

Both robots.txt and robots meta tags rely on cooperation from the robots, and are by no means guaranteed to work for every bot. If you need stronger protection from unscrupulous robots and other agents, you should use alternative methods such as password protection. Too many times I’ve seen webmasters naively place sensitive URLs such as administrative areas in robots.txt. You better believe robots.txt is one of the hacker’s first ports of call—to see where they should break into.

Robots.txt works well for:

• Barring crawlers from non-public parts of your website
• Barring search engines from trying to index scripts, utilities, or other types of code
• Avoiding the indexation of duplicate content on a website, such as “print” versions of html pages
• Auto-discovery of XML Sitemaps

At the risk of being Captain Obvious, the robots.txt file must reside in the root of the domain and must be named “robots.txt” (all lowercase). A robots.txt file located in a subdirectory isn’t valid, as bots only check for this file in the root of the domain.

Creating a robots.txt file is easy. You can create a robots.txt file in any text editor. It should be an ASCII-encoded text file, not an HTML file.

Robots.txt syntax

• User-Agent: the robot the following rule applies to (e.g. “Googlebot,” etc.)
• Disallow: the pages you want to block the bots from accessing (as many disallow lines as needed)
• Noindex: the pages you want a search engine to block AND not index (or de-index if previously indexed). Unofficially supported by Google; unsupported by Yahoo and Live Search.
• Each User-Agent/Disallow group should be separated by a blank line; however no blank lines should exist within a group (between the User-agent line and the last Disallow).
• The hash symbol (#) may be used for comments within a robots.txt file, where everything after # on that line will be ignored. May be used either for whole lines or end of lines.
• Directories and filenames are case-sensitive: “private”, “Private”, and “PRIVATE” are all uniquely different to search engines.

Let’s look at an example robots.txt file. The example below includes:

• The robot called “Googlebot” has nothing disallowed and may go anywhere
• The entire site is closed off to the robot called “msnbot”;
• All robots (other than Googlebot) should not visit the /tmp/ directory or directories or files called /logs, as explained with comments, e.g., tmp.htm, /logs or logs.php.

User-agent: Googlebot
Disallow:

User-agent: msnbot
Disallow: /

# Block all robots from tmp and logs directories
User-agent: *
Disallow: /tmp/
Disallow: /logs # for directories and files called logs

What should be listed on the User-Agent line? A user-agent is the name of a specific search engine robot. You can set an entry to apply to a specific bot (by listing the name) or you can set it to apply to all bots (by listing an asterisk, which acts as a wildcard). An entry that applies to all bots looks like this:

User-Agent: *

Major robots include: Googlebot (Google), Slurp (Yahoo!), msnbot (MSN), and TEOMA (Ask).

Bear in mind that a block of directives specified for the user-agent of Googlebot will be obeyed by Googlebot; but Googlebot will NOT ALSO obey the directives for the user-agent of * (all bots).

What should be listed on the Disallow line? The disallow lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).

Examples:

• To block the entire site: Disallow: /
• To block a directory and everything in it: Disallow: /private_directory/
• To block a page: Disallow: /private_file.html
• To block a page and/or a directory named private: Disallow: /private

If you serve content via both http and https, you’ll need a separate robots.txt file for each of these protocols. For example, to allow robots to index all http pages but no https pages, you’d use the robots.txt files as follows, for your http protocol:

User-agent: *
Disallow:

And for the https protocol:

User-agent: *
Disallow: /

Bots check for the robots.txt file each time they come to a website. The rules in the robots.txt file will be in effect immediately once it is uploaded to the site’s root and the bot comes to the site. How often it is accessed varies on how frequently the bots spider the site based on popularity, authority, and how frequently content is updated. Some sites may be crawled several times a day while others may only be crawled a few times a week. Google Webmaster Central provides a way to see when Googlebot last accessed the robots.txt file.

I’d recommend using the robots.txt analysis tool in Google Webmaster Central to check specific URLs to see if your robots.txt file allows or blocks them, see if Googlebot had trouble parsing any lines in your robots.txt file, and test changes to your robots.txt file.

Some advanced techniques

The major search engines have begun working together to advance the functionality of the robots.txt file. As alluded to above, there are some functions that have been adopted by the major search engines, and not necessarily all of the major engines, that provide for finer control over crawling. As these may be limited though, do exercise caution in their use.

Crawl delay: Some websites may experience high amounts of traffic and would like to slow search engine spiders down to allow for more server resources to meet the demands of regular traffic. Crawl delay is a special directive recognized by Yahoo, Live Search, and Ask that instructs a crawler on the number of seconds to wait between crawling pages:

User-agent: msnbot
Crawl-delay: 5

Pattern matching: At this time, pattern matching appears to be usable by the three majors: Google, Yahoo, and Live Search. The value of pattern matching is considerable. Let’s look first at the most basic of pattern matching, using the asterisk wildcard character. To block access to all subdirectories that begin with “private”:

User-agent: Googlebot
Disallow: /private*/

You can match the end of the string using the dollar sign ($). For example, to block URLs that end with .asp:

User-agent: Googlebot
Disallow: /*.asp$

Unlike the more advanced pattern matching found in regular expressions in Perl and elsewhere, the question mark does not have special powers. So, to block access to all URLs that include a question mark (?), simply use the question mark (no need to “escape” it or precede it with a backslash):

User-agent: *
Disallow: /*?*

To block robots from crawling all files of a specific file type (for example, .gif):

User-agent: *
Disallow: /*.gif$

Here’s a more complicated example. Let’s say your site uses the query string part of the URLs (what follows the “?”) solely for session IDs, and you want to exclude all URLs that contain the dynamic parameter to ensure the bots don’t crawl duplicate pages. But you may want to include any URLs that end with a “?”. Here’s how you’d accomplish that:

User-agent: Slurp
Disallow: /*? # block any URL that includes a ?
Allow: /*?$ # allow any URL that ends in a ?

Allow directive: At this time, the Allow directive appears to only be supported by Google, Yahoo, and Ask. Just as it sounds, it works the opposite of the Disallow directive and provides the ability to specifically call out directories or pages that may be crawled. This may be beneficial after large sections or the entire site has been disallowed.

To allow Googlebot into only the “google” directory:

User-agent: Googlebot
Disallow: /
Allow: /google/

Noindex directive: As mentioned above, this directive offers benefits in eliminating snippetless title-less listings from the search results, but it’s limited to Google. Its syntax exactly mirrors Disallow. In the words of Matt Cutts:

“Google allows a NOINDEX directive in robots.txt and it will completely remove all matching site URLs from Google. (That behavior could change based on this policy discussion, of course, which is why we haven’t talked about it much.)”

Sitemap: An XML sitemap file can tell search engines about all the pages on your site, and optionally, to provide information about those pages, such as which are most important and how often they change. It acts as an auto-discovery mechanism for the spider to find the XML sitemap file. You can tell Google and other search engines about your Sitemap by adding the following line to your robots.txt file:

Sitemap: sitemap_location

The sitemap_location should be the complete URL to the Sitemap, such as: http://www.example.com/sitemap.xml. This directive is independent of the user-agent line, so it doesn’t matter where you place it in your file. All major search engines support the Auto-Discovery Sitemap protocol, including Google, Yahoo, Live Search, and Ask.

While auto-discovery provides a way to inform search engines about the sitemap.xml file, it’s also worthwhile verifying and submitting sitemaps directly to the search engines through each of their webmaster consoles (Google Webmaster Central, Yahoo Site Explorer, Live Search Webmaster Center).

More about Google’s bots

Google uses several different bots (user-agents). The bot for web search is Googlebot. Google’s other bots follow rules you set up for Googlebot, but you can set up additional rules for these specific bots as well. Blocking Googlebot blocks all bots that begin with “Googlebot”.

Here’s a list of Google robots:

• Googlebot: crawls pages from web index and news index
• Googlebot-Mobile: crawls pages for mobile index
• Googlebot-Image: crawls pages for image index
• Mediapartners-Google: crawls pages to determine AdSense content, only crawls sites if show AdSense ads
• Adsbot-Google: crawls to measure AdWords landing page quality, only crawls sites that use Google AdWords to advertise

You can block Googlebot entirely by using:

User-agent: Googlebot
Disallow: /

You can allow Googlebot, but block access to all other bots:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

Issues with robots.txt

Pages you block by using robots.txt disallows may still be in Google’s index and appear in the search results — especially if other sites link to them. Granted, a high ranking is pretty unlikely since Google can’t “see” the page content; it has very little to go on other than the anchor text of inbound and internal links, and the URL (and the ODP title and description if in ODP/DMOZ.) As a result, the URL of the page and, potentially, other publicly available information can appear in search results. However, no content from your pages will be crawled, indexed or displayed.

To entirely prevent a page from being added to a search engine’s index even if other sites link to it, use a “noindex” robots meta tag and ensure that the page is not disallowed in robots.txt. When spiders crawl the page, it will recognize the “noindex” meta tag and drop the URL from the index.

Robots.txt and robots meta tag conflicts

If the robots.txt file and robots meta tag instructions for a page conflict, bots follow the most restrictive. More specifically:

• If you block a page with robots.txt, bots will never crawl the page and will never read any robots meta tags on the page.
• If you allow a page with robots.txt but block it from being indexed using a robots meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.

While robots.txt files are to protect content on a site from being indexed, including a robots.txt file regardless is recommended as many robotic processes look for them and offering one can only expedite their procedures. Together, robots.txt and robots meta tags give you the flexibility to express complex access policies relatively easily:

• Removing an entire website or part of a website.
• Avoiding indexation of images in Google Image Search and other image engines.
• Avoiding indexation of duplicate content on a site.
• Removing individual pages on a site using a robots Meta tag.
• Removing cached copies and snippets using a robots Meta tag.

Both robots.txt and robots meta tag rely on cooperation from the robots, and are by no means guaranteed to work for every robot. If you need stronger protection from robots and other agents, you should use alternative methods such as password protection.

Welcome back from Part 1, where I discussed in detail how to implement URL rewriting with Apache’s mod_rewrite module — complete with example rewrite rules, the more common regular expressions and how to use them. If you recall, I was just starting to get into rewrite rules for 301 redirects using the [R=301] flag. (Incidentally, I much prefer using the RewriteRule directive for setting up my 301 redirects rather than Redirect, RedirectPermanent, or RedirectMatch; more on that later.)

There’s another handy directive I often use in conjunction with RewriteRule, called RewriteCond. You would use RewriteCond if you’re trying to match on something in the query string, the domain name, or other things not present between the domain name and the question mark in the URL (which is what RewriteRule looks at). Note that neither RewriteRule nor RewriteCond can access what is in the anchor part of a URL, i.e. whatever follows a #, because that is used internally by the browser and is not sent to the server as part of the request. The following RewriteCond example looks for a positive match on the host name before it will allow the rewrite rule that follows to be executed:

RewriteCond %{HTTP_HOST} !^www\.example\.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]

Let’s deconstruct what’s happening here. For any host name other than www.example.com, a 301 redirect is issued to the equivalent canonical URL on the www subdomain. The [NC] flag makes the rewrite condition case-insensitive. Where is the [QSA] flag so that the query string is preserved, you might ask? It’s not needed when redirecting; it’s implied.

If you don’t want a query string retained on a rewrite rule with a redirect, put a question mark at the end of the destination URL in the rule. Like so:

RewriteCond %{HTTP_HOST} !^www\.example\.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1? [L,R=301]

Note the exclamation point at the beginning of the regular expression. That is interpreted as “NOT” by the rewrite engine.

Why didn’t I use ^example\.com$ instead? Consider:

RewriteCond %{HTTP_HOST} ^example\.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1? [L,R=301]

Because that wouldn’t have matched on typo domains such as exampel.com that the DNS server and virtual host would be set to respond to (assuming that misspelling was a domain you registered and owned).

Under what circumstances might we want to omit the query string from the redirected URL, as was done in the last two examples? When a session ID or a tracking parameter (like “source=banner_ad1″) needs to be dropped. Retaining a tracking parameter after the redirect is not only unnecessary (because the original URL with the source code appended would have been recorded in your access log files as it was being accessed), it’s undesirable from a canonicalization standpoint. What if you wanted to drop the tracking parameter from the redirected URL, but retain the other parameters in the query string? Here’s how you’d do it, for static URLs:

RewriteCond %{QUERY_STRING} ^source=[a-z0-9]*$
RewriteRule ^(.*)$ /$1? [L,R=301]

and for dynamic URLs:

RewriteCond %{QUERY_STRING} ^(.+)&source=[a-z0-9]+(&?.*)$
RewriteRule ^(.*)$ /$1?%1%2 [L,R=301]

Need to do some fancy stuff with cookies before redirecting the user? Invoke a script that cookies the user then 301s them to the canonical URL:

RewriteCond %{QUERY_STRING} ^source=([a-z0-9]*)$
RewriteRule ^(.*)$ /cookiefirst.php?source=%1&dest=$1 [L]

Note the lack of a [R=301] flag above. That’s on purpose. No need to expose this script to the user. Use a rewrite and let the script itself send the 301 after it has done its work.

Other canonicalization issues worth correcting with rewrite rules and the [R=301] flag include when the engines indexes: 1) online catalog pages under HTTPS URLs, and 2) URLs missing a trailing slash that should be there. First the HTTPS fix:

# redirect online catalog pages in the /catalog/ directory if HTTPS
RewriteCond %{HTTPS} on
RewriteRule ^catalog/(.*) http://www.example.com/catalog/$1 [L,R=301]

Note that if your secure server is separate from your main server, you can skip the RewriteCond line above.

Now to append the trailing slash:

RewriteRule ^(.*[^/])$ /$1/ [L,R=301]

WordPress handles missing trailing slashes by default. Yay WordPress!

Speaking of WordPress, did you know that when you update the “post slug” on a published post (i.e. revise the URL), WordPress will automatically 301 redirect all requests for the previous URL to the new URL? In fact, if you modify the post slug multiple times, all previous iterations will be responded to with a 301! And there won’t be a series of 301s daisy chained together; there is just one redirect issued to the latest iteration. Thus, you can employ a continuous improvement approach to your URL optimization, employing the “thin slicing” methodology I described in a recent column and my SEO Title Tag plugin to mass edit all your permalink post URLs and let WordPress handle the 301s automagically. It’s a beautiful thing.

After completing a URL rewriting project to migrate from dynamic URLs to static, you’ll want to phase out the dynamic URLs not just by replacing all occurrences of the legacy URLs on your site, but also by 301 redirecting the legacy dynamic URLs to their static equivalents. That way, any inbound links pointing to the retired URLs will end up leading both spiders and humans to the correct new URL — thus ensuring the new URLs are the ones that are indexed, blogged about, linked to, and bookmarked. Generally, here’s how you’d accomplish that:

RewriteCond %{QUERY_STRING} id=([0-9]+)
RewriteRule ^get_product\.php$ /products/%1.html? [L,R=301]

However, you’ll get an infinite loop of recursive redirects if you’re not careful. One quick-and-dirty way to avoid that situation is by adding a nonsense parameter to the destination URL for the rewrite and ensuring this nonsense parameter isn’t present before doing the redirect. Specifically:

RewriteCond %{QUERY_STRING} id=([0-9]+)
RewriteCond %{QUERY_STRING} !blah=blah
RewriteRule ^get_product\.php$ /products/%1.html? [L,R=301]

RewriteRule ^products/([0-9]+)/?$ /get_product.php?id=$1&blah=blah [L]

Notice above that I used two RedirectCond lines, stacked on top of each other. All redirect conditions listed together in the same block will be “ANDed” together. If you wanted the conditions to be “ORed”, it would require the use of the [OR] flag.

Enough about redirects. Let’s move on to lookup tables and RewriteMap, a directive that functions within your server config file (not .htaccess). Let’s say you’d like to rewrite URLs that contain ID numbers to URLs that contain keywords. A laudable goal. Now let’s say you don’t have the lookup table in your database. You could reference a flat file — in text, or in DBM format (which is faster) — containing your mappings of ID numbers to keywords using RewriteMap, then base your RewriteRule on data found in that flat file. Here’s a hypothetical lookup table:

canon-g10-digital-camera /get_product.php?id=1001&blah=blah
128-gig-ipod-classic /get_product.php?id=1002&blah=blah

And here’s what the corresponding RewriteMap and RewriteRule directives might look like:

RewriteMap prodmap txt:/home/someusername/prodmap.txt
RewriteRule ^/products/(.+)\.html$ ${prodmap:$1} [L]

Conversely, you’ll want to 301 all of the legacy ID-containing URLs to the new keyword-containing ones. Like so:

RewriteMap prodmap2 txt:/home/someusername/prodmap2.txt
RewriteCond %{QUERY_STRING} id=([0-9]+)
RewriteCond %{QUERY_STRING} !blah=blah
RewriteRule ^get_product\.php$ ${prodmap2:%1}? [L,R=301]

The corresponding lookup table for the above (”prodmap2.txt”) would look something like:

1001 /products/canon-g10-digital-camera.html
1002 /products/128-gig-ipod-classic.html

You aren’t restricted to text or DBM files. You could alternatively install a script that looks up what the rewrite rule had captured into memory (between the parentheses) and then delivers back to the rewriting engine the corresponding destination. Here’s the slightly modified RewriteMap for such an instance:

RewriteMap prodmap prg:/home/someusername/mapscript.pl
RewriteRule ^/products/(.+)\.html$ ${prodmap:$1} [L]

On to a different problem. Let’s say you wanted to rewrite to a URL located on another server. You can do it with the [P] flag. The “P” stands for “proxy”. For example, you could proxy the Google home page on your server (sans images) with the following rewrite rule:

RewriteRule /google\.html$ http://www.google.com/ [P,L]

Without the [P] flag, the rewrite rule above would behave like a redirect.

You might be wondering how to accomplish all this wizardry if your server is running Microsoft IIS Server instead of Apache. As I mentioned in Part 1, the rules don’t differ greatly between mod_rewrite and ISAPI_Rewrite. For instance, instead of initializing things with “RewriteEngine on”, you would specify “[ISAPI_Rewrite]” on the first line of the httpd.ini file. Instead of [R=301], you would use [RP] to issue a 301. Instead of [NC] for case insensitivity, you would use [I]. And so on. The easiest way to convey this is through some illustrative examples:

#Capitalization and IIS' case insensitivity with regard to URLs
RewriteRule (.*) http://www.example.com$1 [I,RP,L]

#Add trailing slash if it's missing
RewriteCond Host: (.*)
RewriteRule ([^.?]+[^.?/]) http\://$1$2/ [I,RP,L]

At the start of this article, I promised we’d revisit my reasoning as to why rewrite rules are my preferred method of redirecting. It’s simply because RewriteRule is so darned powerful and flexible in comparison to Redirect, RedirectPermanent and RedirectMatch. Even though RedirectMatch supports regular expressions, it doesn’t offer nearly as comprehensive of a feature set as RewriteRule and RewriteCond. However, you may be on a web host or server that doesn’t have mod_rewrite installed/enabled. If that’s the case, it may be helpful to see a few examples of these alternative directives, which can be used in either .htaccess or httpd.conf:

# 301 an individual URL
Redirect 301 /old_url.htm http://www.example.com/new_url.htm

# drop the index.html off the end of subdirectories and 301
RedirectMatch 301 ^/(.+)/index\.html$ http://www.example.com/$1/

That’s all I’ve got — for now. I promise in my next article I won’t geek out so much. If you stayed with me this whole time, you deserve a cookie. Hit me up at the next conference and I’ll swipe one for you out of the speaker lounge.

If you’re dealing with a large complex website, rewriting your URLs from dynamic to static and placing all the necessary 301 redirects in place is – as programmers would say, nontrivial. The devil is in the details. Granted, the redirecting piece may not be quite as onerous anymore thanks to the advent of the search engines’ new canonical tag, but you still have to worry about the user experience for users coming in to the obsolete URLs (through bookmarks, through old links, etc.). Therefore, ideally you’ll still want the 301s in place. Regular expressions and mod_rewrite (or alternatively, ISAPI_Rewrite for IIS) to the rescue!

Last week, when I presented on the 301 Redirect panel at SMX West, I gave folks a view “under the hood” at accomplishing redirects using rewrite rules and regular expressions – complete with the code necessary to pull it off. It wasn’t for the faint of heart. (My Powerpoint can be downloaded here.)

Regular expressions are so complex there are entire books dedicated to the topic (such as the excellent Mastering Regular Expressions, 3rd Ed. by Jeffrey E. F. Friedl and published by my long-time favorite book publisher, O’Reilly). Before we delve into the use of regular expressions in rewrite rules, however, let’s step back and look at the URL rewriting process.

The three types of URL rewrites

Rewriting of search engine sub-optimal URLs can be accomplished through three approaches. The first of which – using a “URL rewriting” server module/plugin such as mod_rewrite for Apache or ISAPI_Rewrite for Microsoft IIS Server – is the most popular. If you can’t use a URL rewriting module on your server, you might recode your scripts to extract variables out of the “path_info” part of the URL instead of the “query_string”. An example of this might look like http://www.example.com/index.php/category/widgets.

With either approach, you’d want to replace all occurrences of your old URLs in links on your site with your new search-friendly URLs. Additionally, you may wish to 301 redirect the old URLs to the new ones, but this is apparently optional with the advent of the canonical tag. The third approach would be to use a proxy server based solution (e.g. GravityStream) that eliminates the need to recode your site or re-architect your CMS/e-commerce platform. This can be useful when IT department involvement with SEO projects must be minimized, for whatever reason.

Let’s assume you’re going with first approach – utilizing a rewriting module. If you are running Apache as your web server, you would place “rules” within your .htaccess file or your Apache configuration file (e.g. httpd.conf or the site-specific config file in the sites_conf directory). Similarly, if you are running IIS Server, you’d use an ISAPI plugin such as ISAPI_Rewrite and place rules in an httpd.ini config file. Note that rules can differ slightly on ISAPI_Rewrite compared to mod_rewrite. For Apache and mod_rewrite, your .htaccess would start off with:

RewriteEngine on
RewriteBase /

Note that you should omit the second line above if adding the rewrites to your server config file, since RewriteBase is not supported there, only in .htaccess. We’re using RewriteBase above so that we won’t have to have “^/” at the beginning of all the rules, just “^”.

After this comes the rewrite rules. Let’s say we wanted to have requests for product page URLs of the format http://www.example.com/products/123 to display the content found at http://www.example.com/get_product.php?id=123, without the URL changing in the Location bar of the user’s browser and without you having to recode the get_product.php script. (Of course this doesn’t replace all occurrences of dynamic URLs within the links contained on all the site pages; that’s a separate issue.) Accomplishing this can be done with a single rewrite rule, like so:

RewriteRule ^products/([0-9]+)/?$ /get_product.php?id=$1 [L]

In the above example, ^ signifies the start of the URL following the domain, $ signifies the end of the URL, [0-9] signifies a digit and the + immediately following it means one or more occurrences of a digit. Similarly, the ? immediately following the / means zero or one occurrences of a slash character. The () puts whatever is wrapped within it into memory. You can then access what’s been stored in memory with $1 (i.e. what is in the first set of parentheses). Not surprisingly, if you included a second set of parentheses in the rule, you’d access that with $2. And so on. The [L] flag saves on server processing by telling the rewrite engine to stop if it matched on that rule. Otherwise all the remaining rules will be run as well.

Sound complicated? You ain’t seen nothin’ yet! Here’s a slightly more complex example, where URLs of the format http://www.example.com/webapp/wcs/stores/servlet/ProductDisplay?storeId=10001&catalogId=10001&langId=-1&categoryID=4&productID=123 would be rewritten to http://www.example.com/4/123.htm:

RewriteRule ^([^/]+)/([^/]+)\.htm$ /webapp/wcs/stores/servlet/ProductDisplay?storeId=10001&catalogId=10001&langId=-1&categoryID=$1&productID=$2 [QSA,L]

The [^/] signifies any character other than a slash. That’s because, within square brackets, ^ is interpreted as “not”. The [QSA] flag above is for when you don’t want the query string dropped (like when you want a tracking parameter preserved).

To write good rewrite rules you will need to become a master of “pattern matching” (which is simply another way to describe the use of regular expressions). Let’s look at some of the most important special characters and how they are interpreted by the rewrite engine:

* means 0 or more of the immediately preceding character + means 1 or more of the immediately preceding character ? means 0 or 1 occurrence of the immediately preceding character ^ means the beginning of the string $ means the end of it . means any character (i.e. wildcard) \ “escapes” the character that follows, e.g. \. means dot [ ] is for character ranges, e.g. [A-Za-z] for any lower or upper case letter ^ inside [] brackets means “not”, e.g. [^/] means not slash

It’s incredibly easy to make errors in regular expressions. Some of the common gotchas that lead to unintentional sub-string matches include:

• using .* when you should be using .+ since .* can match on nothing
• not “escaping” with a backslash special characters that you don’t want interpreted, like when you specify . instead of \. and you really meant a dot rather than any character. (thus default.htm would match on defaultshtm)
• omitting ^ or $ on the assumption that the start or end is implied (thus default\.htm would match on mydefault.html whereas ^default\.htm$ would only match on default.htm)
• using “greedy” expressions that will match on all occurrences rather than stopping at the first occurrence.

What do I mean by “greedy”? The easiest way to explain it is to show you an example. Let me illustrate:

RewriteRule ^(.*)/?index\.html$ /$1/ [L,R=301]

will redirect requests for http://www.example.com/blah/index.html to http://www.example.com/blah//. Probably not what was intended. Why did this happen? Because .* will capture the slash character within it before the /? gets to see it. Thankfully, there’s an easy fix. Simply use [^ or .*? instead of .* to do your matching. For example, use ^(.*?)/? instead of ^(.*)/? or [^/]+/[^/] instead of .*/.*

So, to correct the above rule you could use the following:

RewriteRule ^(.*?)/?index\.html$ /$1/ [L,R=301]

Why wouldn’t you use the following?

RewriteRule ^([^/]*)/?index\.html$ /$1/ [L,R=301]

Because it would only match on URLs with one directory. URLs containing multiple subdirectories such as http://www.example.com/store/cheese/swiss/wheel would not match.

And why wouldn’t you use the following?

RewriteRule ^(.*)index\.html$ $1/ [L,R=301]

Because it would match on http://www.example.com/myindex.html as well (since it’s not specified that the character immediately preceding index must be a slash).

Does your head hurt yet? As you might imagine, testing/debugging is a big part of URL rewriting. When debugging, the RewriteLog and RewriteLogLevel directives are your friend! Set the RewriteLogLevel to 4 or more to start seeing what the rewrite engine is up to when it interprets your rules.

By the way, the [R=301] flag in the last few examples above — as you might guess — tells the rewrite engine to do a 301 redirect instead of a standard rewrite.

Stay tuned for my column next month for Part 2, where I cover the all-important RewriteCond directive, lookup tables, ISAPI rewrite rules, proxying, alternatives to RewriteRule for redirecting, and more.