Big SEO: Automate Or Die Trying

If you are one of the growing numbers of in-house SEOs working on larger web sites, things can get a bit “scratch-your-head, where-do-I-start” difficult. If you’re managing tens of millions of pages, or even tens of thousands of pages, there’s no doubt you need to automate things. Even those working on smaller sites that produce a large volume of unique content each month could benefit from automation.

This doesn’t mean buying some “search optimization software,” though. Rather, it entails developing a completely holistic view of the entire SEO process, including your:

  • Search optimization efforts
  • Corporate goals
  • Projected ROI for every step along the way
  • Timelines
  • In-house programming capacity and more than one glance at your CMS (content management system)

In the end, you’re looking for the best way to cover the basics of on-page optimization, while being able to manage this effort across those millions of pages.

Some very quick math shows why you must go down this road:

Time to optimize one page = 4 minutes (writing unique tags, updating code, publishing the page, etc.) Number of pages = 500,000

Number of hours needed = 33,333, or 1,389 days (3.8 years)

Not that this is a hard thing to wrap our heads around, but it’s obviously clear that manually touching this volume of pages is unrealistic.

So, what to do?

Build some proprietary tools

Many may cringe at this idea, and for good reason. Building your own tool set to manage all your optimization needs is no simple task. You’ll need to cover items like:

  • A way to automate the insertion of meta tag data based on the actual content of a given page
  • A way to manually adjust any one page
  • A way to test the system against pre-defined rules to give you a red light/green light report to help find and drill into any problem areas

Here are some steps to help you tackle the bigger points:

Step 1: How big is this site, anyway? You’ll need a realistic picture of how large the site is currently, and its projected growth over the next few months or so. Break this out by folders or subdomains, however the site is naturally divided.

Step 2: What matters to you? Decide what should be covered with your basic on-page optimization. I cover things like the <title> tag, the <description> tag, and the <keywords> tag. Yes, I still feel there’s value in filling in the keyword tag properly (not much, but why do only part of a job?). I also target <doctype> data.

This is only the basic meta tag stuff, so what’s next?

Step 3: Your CMS – friend or foe? That long look at your CMS comes next, because how it builds things like the page itself and the URL can play a role in your search efforts.

As content is built, you’ll need a way to ensure select items are “tagged” on a page, so your automated system knows where to find the data it needs to do its job. “Tagged” does not mean tagging on blogs and on social sites – we’re talking about a way to identify a particular keyword or phrase here. You may choose to tell your system that anything that appears in the <H1> tags is the data the system will use.

It’s also important to look at the structure of the URL your CMS builds for any given page. Are there characters in there that cause spiders to pause? If so, the page may not get crawled. Not crawled means not indexed, so it’ll never rank in the results pages. Worse, some URLs are so long that even though they can be crawled, they won’t be due to their length.

As well, since this is a game of inches, not miles, I’m still a fan of integrating relevant keywords into URLs where you can. Many scoff at this these days, but if Google goes to the trouble of bolding the keywords/phrases a searcher requested in its results, why not include the keywords in your URL—it’s one more chance to stand out in a crowded SERP page.

Step 4: Educate those who build the content. You will need to touch the humans who build your content at some point. They will need to understand how what they write affects the results in search, months and years down the road. Providing them with a suggested keyword list is a great way to help them understand where you are going. Far from telling a writer what to write, this is simply a means of offering them a choice. If using the word “car” is better suited to the phrase you want the page to rank well for, suggest they use “car” near the top of their article, then work in “auto” or “automobile” later in the article for readability.

Step 5: Pitching the value. Moving from concept to reality is a big step here. You may potentially need hundreds of hours of programming time not only to build the tool set you’re after, but to tweak any templates currently being used by your site, and to rework areas of the CMS itself to allow it to make the changes you need on the fly.

Getting a system like this built will take a company-wide commitment, so be ready to show the ROI you expect to achieve. If you’re one of the lucky ones whose company is currently shopping for a CMS, get your own programmers to wrap their heads around your needs in the search world. They can then help evaluate the CMS with an eye towards integrating these proprietary tools into the mix.

Another option is to build it all from scratch—CMS and all.

In the end, big SEO needs dedicated tools to help manage the workload and get things done in a timely manner. Having tens of thousands of pages with no <title> tags is not the answer. You need a system that’s smart enough to know what to do on its own (with a basic amount of original input by you, of course), that you can over-ride manually when you need to, and that can serve up reports on its activities to help you quickly find and fix trouble spots.

Expect projects like this to take months to get up and running. If your company has a large website, it’s critical to take a moment to do a real gut-check. If you’re not managing things this large in an automated manner, you’re making it easy for your competitors to steal your thunder.

The bottom line, which will ultimately determine what you end up with, is: how committed are you to growing inbound search traffic?

Duane Forrester is an in-house SEM, sits on the Board of Directors with SEMPO and can be found at his blog where he speaks about online marketing and monetizing websites. The In House column appears on Wednesdays at Search Engine Land.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: SEO | In House Search Marketing | SEM Tools | SEO: General


About The Author: is an in-house SEM with Microsoft, is a former Board of Directors member with SEMPO, can be found at his blog where he speaks about online marketing and monetizing websites and is the author of two books: How To Make Money With Your Blog & Turn Clicks Into Customers.

Connect with the author via: Email


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.


Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide