Optimizing A Large-Scale Web Site? Do The Two-Step

The need for keyword research is pretty much a given. But applying that research on a massive scale is where things usually break down. If you’re dealing with 100,000 web pages, you don’t have time to touch every page. How can one do this scalably? The answer: programmatic optimization.

In a previous article “Scalable On-Page SEO Strategies,” I discussed various approaches to optimizing large-scale web sites. In the article I alluded to creating “recipes” or formulas for optimized title tags, and allowing that automation to be overridden by manual optimization. Let’s look at this process in more detail, and applied not just to page titles but other HTML tags as well.

Programmatic formulas

To raise the content optimization of every page on the site to a 60-75% optimal level, first create formulas for the title tag, H1 heading, meta description and (optionally) the meta keywords that correspond to data fields in the database. Each template will have different data fields available to it, so each template needs its own formula. Here is a simple example from Amazon.com:

Subcategory Template: Business & Investing Books (http://www.amazon.com/Business-Investing-Books/b?ie=UTF8&node=3)

Title tag: {subcategory name} {category name} at Amazon.com

H1 Heading: {subcategory name} {category name}

Meta Description: Earth’s Biggest Selection of {subcategory name} {category name}. Amazon.com strives to be Earth’s most customer-centric company, where customers can find anything they might want to buy online at the lowest possible prices.

Meta Keywords: {subcategory name}, {category name}, amazon.com

Manual Review

Programmatic optimization is the first step to large-scale content optimization. But then a second pass is required.

For pages that drive high traffic and revenue (or that have the potential to be), manual content optimization should be done to target the more competitive keyword terms and phrases in your keyword list.

To make that keyword list actionable, you’ll need keyword to URL pairings. In other words, you’ll need to map your desired keywords to URLs on your site. It’s possible to do this algorithmically (automatically), though a solution that can do this well isn’t going to be cheap. Unless you have an enterprise-sized budget, probably you’ll be going with a manual approach.

Here is a manual approach that works well. Create a spreadsheet that contains every page you want to manually optimize. Paste in the keyword research you’ve already done, terms and popularity data. Then place one keyword next to every page — this is your primary keyword for the page, the keyword that you will target content optimization around to send the strongest signal to the search engines. Highest popularity keywords should be matched with the strongest pages, usually the pages at the highest levels of your site. If you still have keywords left, match the remaining keywords to pages as secondary keywords. These should support the keyword theme for the primary keyword. We refer to the primary keyword as the “declared search term.”

Once the keywords are mapped to the pages you plan to optimize manually, all that’s left is the writing. Easier said than done.

Long tail strategies

For this, consider a “Long Tail” SEO approach that we at Netconcepts call “thin slicing” — a term and concept popularized by Malcolm Gladwell in his best selling book Blink. Gladwell uses “thin slicing” in the context of “rapid cognition,” where one makes snap judgments in their field of expertise. Surprisingly, those snap judgments are often times more accurate than considered opinion, i.e. assessments that have been labored over. There’s one important caveat: the phenomenon only holds true for experts, not for amateurs.

Thin slicing in the context of SEO involves touching key elements such as title tags, keyword URLs, H1 headings, and meta descriptions across thousands of pages quickly, monitoring for impact, and refining based on those results. Don’t try to optimize everything on the page, or you’ll get bogged down and lower your productivity.

We believe that thin slicing is as effective for the SEO expert as it is for Gladwell’s art historian. This may sound heretical, but with a daunting number of pages to get through, deciding on synonyms, verb tenses and word order should rely more on your intuition and a less on your keyword research. Make quick decisions; don’t overthink it or overanalyze it. Logically, by agonizing over the very best string of words to use for a title or H1 and continuously consulting keyword research tools, you get caught in the weeds and thus it costs you opportunity. After all, it’s holding you up from getting to the remaining thousands.

The “How” for thin slicing depends very much on your web site’s infrastructure. For example, a WordPress based site can employ the “mass edit” admin capability of our free SEO Title Tag plugin. It supports optimizing title tags and URLs (post “slugs”, more accurately) across many posts and pages quickly, without having to go to each post’s Edit screen individually.

Then there’s the “bulk upload” approach, where you import an updated list of title tags (or H1s or whatever) into your website’s underlying database. You start with a database export in CSV (comma separated values) format of your current title tags — along with the corresponding item ID numbers for each record, of course. Load the CSV file into Microsoft Excel and do your title tag optimization in the spreadsheet. Then upload the optimized title tags back into the database. Note that if your database does not have a field for the title tag, you’ll have to create it and re-code your site to override the programmatic title with the contents of this new field when it is populated with data. Rather than having to maneuver through phpMyAdmin or rely on your database administrator, have a CSV file upload function built into the admin interface of your content management system (CMS). When we added this capability to our GravityStream proxy admin, our optimizers and those at our clients and partner resellers experienced a nice boost in productivity.

If you prefer a forms-based web interface over working in Excel, you could build a “mass edit” view into your content management system’s admin interface, similar to the “mass edit” interface in SEO Title Tag. One feature we found invaluable when using web forms for thin slicing was to make the number of rows displayed per page user-configurable. Some users will want to display hundreds of records per screen, others will want much fewer, as too big of a web page will cause their web browser to crash or time out.

Next time you feel like optimizing 100,000 pages, remember the Two-Step: programmatic optimization first, then manual content optimization second.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: All Things SEO Column | Channel: SEO | How To: SEO


About The Author: is the author of Google Power Search, creator of the Science of SEO and co-author of The Art of SEO now in its second edition, both published by O'Reilly. Spencer is also the founder of Netconcepts and inventor of the SEO technology platform GravityStream. He also blogs on his own site, Stephan Spencer's Scatterings.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • gandalf94305

    Manual keyword assignment for a large-scale website? You must be kidding to make the a real recommendation… I know it is the way how people do it today, however, I would have had expected a bit more innovation from a posting here.

    The only way you can really manage SEO in a large-scale website that may even have significant dynamic parts is by support of automatic entity extraction and automatic assignment using a recommendation engine. There may be few manual optimizations as you need them, but in comparison to the volume of pages, this would be a smaller fraction.

    Manual approaches fail because you won’t be able to easily find all keyword candidates in the first place. Then, assigning the keywords in a consistent manner and accomodating constant changes to pages and contents is the second challenge. Third, large-scale changes because of new campaigns or newly discoverey hype topics won’t be easy to apply.

  • Red_Mud_Rookie

    It’s a good idea to introduce multiple combinations for the formulaic approach to titles and descriptions.
    If every page is:

    Earth’s Biggest Selection of {subcategory name} {category name}. Amazon.com strives to be Earth’s most customer-centric company, where customers can find anything they might want to buy online at the lowest possible prices.

    …then it’s more likely to start looking like duplicate content. The reason I think this is worth a mention is that on page content done in this manner WILL get downgraded by Google – I have seen it happen.


Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide