Say you want a list of the Fortune 500 companies’ websites. Normally, this would be an arduous task. CNN’s Fortune magazine website links to the websites, but you have to visit every company profile on a separate page. Watch what I do with a spreadsheet and a macro.

Get a list of company page URLs from the CNN Money Fortune 500 list.

I use the Web Developer tool to get a list of links to all the company profiles. Just select Information and View Link Information.

View links on a webpage

Then, copy the links you want to scrape.

A list of webpage links

And, paste into Excel.

A list of webpage links in Excel

Add these formulas to row 1:

Column B: 1 You will assign a unique number to each row. This will be used to name your download files.

Column C: =”URL GOTO=” & A1 This is the command that will tell the macro what web page to visit.

Column D: =”SAVEAS TYPE=HTM FOLDER=c:\web-pages FILE=” & B1 & “.html” This saves a copy of the web page on your hard drive.

Column E: WAIT SECONDS=3 This instruction tells the macro to pause for three seconds. I find a brief pause between pages helps keep the macro running smoothly.

In Excel, number column B 1 to n (1, 2, 3, 4, etc.).

Copy row 1 of columns C, D, and E then paste them into every row so your spreadsheet looks like this:

A Web browser macro in Excel

You are now done with Excel and ready to create your macro. First, each cell has to become a separate cell. Below is my trick for accomplishing this.

Copy the macro in Excel, columns C, D, and E; only the completed cells, no empty cells.

Paste special as text into Word.

Formatting the Excel Web browser macro in Word

In Word, use Find and Replace. Find ^t, tabs, and replace with ^p, paragraph breaks. Replace All. Your macro will change to one line per instruction.

The formatted macro in Word

Before you go further, you will need to create your download folder. In this tutorial, it is

C:/web-pages

Install iMacros for Firefox. Avoid using the Chrome and other browser versions as they lack important features. After installing, you will find the program in your View Sidebars menu.

iMacros for Firefox

Copy your macro from Word.

In iMacro, open #Current.imm, or any macro. It does not matter because you will give your macro its own name. Select Edit then Edit Macro. Paste your macro into the editor. Instead of the Save button, click on the disk icon – . Name your macro #Fortune.imm.

The iMacros editor

Go back to the iMacro menu. Select Edit, Refresh Macro List.

Click on #Fortune.imm in the macro list. Select Play, Play (the big button).

Your macro will run and download all the pages into C:/web-pages.

And there you have it. This tutorial shows three tricks I use:

• Using macros to automate repetitive tasks
• Using Excel formulas to write my macros
• Using Word to convert tabular Excel content into separate lines

I admit I am leaving you hanging because, at this point, you have a set of HTML files on your hard drive, but you do not yet have a list of links to the actual Fortune 500 websites. To automate that part, I use a PHP script that opens each HTML file and extracts the URLs using PHP’s pattern matching command.

I’ll share my PHP script in my next column.

I use macros to capture all sorts of information, including SEO tools data for multiple sites and even Google keyword tool suggestions. Macros make it a breeze to dig through lots of pages, and Excel makes the creation of macros almost effortless.

As my friends know, in January, I became an in-house SEO. Now that I have a few weeks under my belt, I thought I’d discuss some of the differences between agency SEO and in-house SEO, the type of traits it takes to succeed in-house, and some of the traps you can land in.

Agency vs. In-house

Let’s start with some of the differences between agency SEO and in-house SEO.

In-house Has No Hamster Wheel

In an agency, it’s inevitable. While waiting for clients to implement your recommendations, you hit roadblocks and dead ends.

Is this the perfect workout for some SEO agency employees? Image Credit: PBoyd04

Except, you have a minimum number of billable hours and a table declaring how many recommendations you must produce. You cannot start a new project without a client meeting. The next meeting is scheduled in two weeks, when you must present your recommendations from today. Frustrated, you look for any suggestion you can give, whether it will actually help or not.

As an in-house SEO, I can investigate and negotiate roadblocks or I can find something else to work on, something that will create value.

No Billable Hours

Saints be praised; no more billable hours. I don’t care what tool or stopwatch you use, tracking billable hours is an impossible farce created by sick, sadistic minds. It always comes down to imperfect memories and estimates.

Image credit: Zak Greant

In-House Has More Distractions

At the agency, the CEO and I were the go-to people for all things search and social. And, because he was the CEO, I got most of the questions. That meant I worked on every client whether I was on the team or not.

Imagine my surprise. As an in-house SEO, I get more questions from colleagues and interruptions. It makes sense. At the agency, everyone from the receptionist to the CEO lives and breathes search and social. They know the 80%. They need help with the difficult 20%.

In-house, I have to educate and evangelize from the ground up. People here know organic search and social are important, but they are just learning how to implement it. Unlike PPC landing pages, which are largely isolated, every brochure page, case study, blog post… every indexed page affects SEO.

In-house, You Can Walk Down The Hall & Face-to-Face

When you hit roadblocks with agency clients, you’re at their mercy. Whether it’s a communications slowdown, a technical hurdle, or defensive posturing, you have to wait for the client to implement or give you the go-ahead.

In-house, I can simply leave my desk and visit my co-workers. I don’t always get the answers I want. However, I can learn exactly where each problem lies and work with people who understand and share my goals.

SEO Communications

Whether you work as an SEO at an agency or inside a business with more than a few employees, there are many things you must be able to communicate, including the following:

Communicate Clearly Without Jargon

When you spend your days with co-workers inside agency walls or with colleagues at conferences, it’s easy to communicate using terms the uninitiated will not understand. A wise professor told me the true meaning of knowledge is the ability to teach. Know your plain language definitions and practice using them. Let colleagues know you want them to ask for an explanation when you say something they do not understand.

Be Disciplined & Concise

Similarly, you cannot monopolize everyone’s time by droning on and on. If you cannot be clear, persuasive, and brief at the same time, you will never succeed as an in-house SEO.

Image Credit: Grotuk

Stay Resolute, Politically Perceptive & Judicious

On day one, my CEO told me our IT director’s prime directive, don’t let the app crash. Smartsheet’s reliability is as important as its features. Because our website and application are on the same domain, certain things require heavy testing or are simply off limits. At the agency, this would have been the end of the discussion. In-house, I can respect these challenges and I get to work with the IT director and application developers to overcome them.

Select High Value Activities, Then Stay Organized & On Track

When I arrived at Smartsheet, I saw many different things I could work on. Throughout my first month, I seemed to explore or pursue a different opportunity every day. During my second month, I began to settle down and isolate things that I can accomplish and will lead to the greatest returns.

Whether your SEO department is just you or a team, you can only accomplish so many projects within so much time. Make the most of what you have.

Knowledge You Can Share

As an in-house SEO, you will enjoy sharing your knowledge. Keep in mind, there are some things worth sharing right away.

The SEO Arms Race

It isn’t enough to optimize your content and earn links or authority. SEO is an arms race. You have to catch-up with the leading keyword competitors and pass them at the same time they accelerate their own content and authority building. It’s not enough to be as good. You must become better, faster and stronger.

The search engines are in an arms race of their own, with Spammers. While the search engines get better at overcoming Spam, the Spammers keep finding new ways to manipulate the search engines. This is an arms race no legitimate business should engage in. Do you want to be the SEO who burns your employer’s domain?

Site Optimization Vs. Page Optimization

Make sure your marketing department understands site optimization concepts like internal link architecture, domain authority, and supporting content.

In a competitive space, you’re not going to publish one page about red widgets and leap into Google’s top ten. It can take several supporting documents, many with their own link and citation attraction qualities. Especially if your company is used to paid search, they may not realize this.

Not Every Page Is An SEO Landing Page

Does everyone understand that while you can optimize any page for either short- or long-tail keywords, not every page is going to rank or drive traffic? It is usually better to keyword optimize your top-level and high-value content, then skip the rest and move on to publishing new content that will earn links and authority.

For the sake of clarity, you will still need to clear-up any technical SEO errors and optimize your internal linking. Also, just because you’re not optimizing a page does not mean you won’t edit it (for example, to add an anchor text link to a real SEO target page).

At some point, after you develop a track record of SEO wins, start revisiting your lower-value pages and check if you can optimize them for long-tail keywords. This is a great task for interns or new staff members who need to build-up their familiarity with your website.

Different Types Of Content Have Different SEO Strength

This is a favorite of mine. Different Web assets or documents serve different purposes and have their own unique SEO strength.

For example, a case study may be ideal to launch a press release campaign around while it is unlikely to go viral on Twitter or earn many links. Live blogging from a popular conference can earn lots of mentions and links, but may not convert many sign-ups or sales. Prepare your company to create many types of content for many different purposes.

This is where I insert the flywheel analogy. The more popular your brand, blog, and social media assets are, the more likely they are to help SEO in different ways. You have to earn that reputation. On a popular blog, a post might go viral, while on a lesser known blog, the exact same post will just sit there. Get your flywheel spinning.

Playing Within Your Competitive Footprint

Be sure your co-workers understand what keywords your company can rank for now and which ones are within reach. There is a reason so few SEO professionals are trying to rank for the keyword SEO, we know it’s a waste of time. Create short-term, long-term, and big, hairy audacious goals. Be assertive and aggressive, but do not create unreasonable expectations.

Here is one of my tips. When I do keyword research, I select based upon the phrase match numbers, but I always report the exact match numbers.

Optimization Vs. Over Optimization

Warn your coworkers about over optimization. There are two good reasons for this. First, you don’t want to incur an over-optimization penalty from the search engines.

Another reason is to give you some breathing room. Everywhere you go, there will be that person, the one who says things like, “You did this over there, so why haven’t you done it over here?” As Bones McCoy might say, “For God sakes Jim, you’re a human being, not a machine.”

Even if you keep checklists (you should), you will not optimize every page exactly the same. SEO is a craft, part science and part art. The more you do it, the better you get, and the more intuitive it becomes. Sometimes, it’s just nice to be able to say in a calm, reassured voice, “I didn’t want to over optimize the page, but I can reconsider that.”

Robots.txt

Robots.txt is a file you place in your website’s top level directory, the same folder in which a static homepage would go. Inside robots.txt, you can instruct search engines to not crawl content by disallowing file names or directories. There are two parts to a robots.txt directive, the user-agent and one or more disallow instructions.

The user-agent specifies one or all Web crawlers or spiders. When we think of Web crawlers we tend to think Google and Bing; however, a spider can come from anywhere, not just search engines, and there are many of them crawling the Internet.

Here is a simple robots.txt file telling all Web crawlers that it is okay to spider every page:

User-agent: *
Disallow:

To disallow all search engines from crawling an entire website, use:

User-agent: *
Disallow: /

The difference is the forward slash after Disallow:, signifying the root folder and everything in it, including sub-folders and files.

Robots.txt is versatile. You can disallow entire sub-folders or individual files. You can disallow specific search engine spiders like Googlebot and Bingbot. The search engines even extended robots.txt to include an Allow directive, file or folder name pattern matching, and XML sitemap locations.

Here is a beautifully executed robots.txt file from SEOmoz:

#Nothing interesting to see here, but there is a dance party
#happening over here: http://www.youtube.com/watch?v=9vwZ5FQEUFg

User-agent: *
Disallow: /api/user?*
Disallow:

Sitemap: http://www.seomoz.org/blog-sitemap.xml
Sitemap: http://www.seomoz.org/ugc-sitemap.xml
Sitemap: http://www.seomoz.org/profiles-sitemap.xml
Sitemap: http://app.wistia.com/sitemaps/2.xml

If you are unfamiliar with robots.txt, be sure to read these pages:

• http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=40367&rd=1
• http://www.bing.com/webmaster/help/how-to-create-a-robots-txt-file-cb7c31ec
• http://www.bing.com/community/site_blogs/b/webmaster/archive/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation.aspx

What robots.txt does not do is to keep files out of the search engine indexes. The only thing it does is instruct search engine spiders not to crawl pages. Keep in mind that discovery and crawling are separate. Discovery occurs as search engines find links in documents. When search engines discover pages, they may or may not add them to their indexes.

Robots.txt Does Not Keep Files Out Of The Search Index!

See for yourself at  site:permanent.access.gpo.gov.

Is Robots.txt A Security Or Privacy Risk?

Using robots.txt to hide sensitive or private files is a security risk. Not only might search engines index disallowed files, it is like giving a treasure map to pirates. Take a look for yourself and see what you learn.

• http://www.google.com/robots.txt
• http://www.bing.com/robots.txt
• http://searchengineland.com/robots.txt

Here is Search Engine Land’s robots.txt file.

User-Agent: *
Disallow: /drafts/
Disallow: /cgi-bin/
Disallow: /gkd/
Disallow: /figz/wp-admin/
Disallow: /figz/wp-content/plugins/
Disallow: /figs/wp-includes/
Disallow: /images/20/
Disallow: /css/
Disallow: /*/feed
Disallow: /*/feed/rss
Disallow: /*?

I used it to search for inurl:http://searchengineland.com/figz. As you can see, I found a few files I am probably not supposed to know about.

Don’t worry; if I had seen something risky or sensitive on Search Engine Land, I would never have shared this example. Can you say the same about your website or online application?

Use Robots <META> Tag To Keep Files Out Of The Search Index

Because robots.txt does not exclude files from the search indexes, Google and Bing follow a protocol which does accomplish exactly that, the Robots <META> tag.

<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
</head>

The robots <META> tag provides two instructions:

1. index or noindex
2. follow or nofollow

Index or noindex instructs search engines whether or not to index a page. When you select index, they may or may not choose to include a webpage in the index. If you select noindex, the search engines will definitely not include it.

Follow or nofollow instructs Web crawlers whether or not to follow the links on a page. It is like adding an rel=”nofollow” tag to every link on a page. Nofollow evaporates PageRank, the raw search engine ranking authority passed from page to age via links. Even if you noindex a page, it is probably a bad idea to nofollow it. Let PageRank flow through to its final conclusion. Otherwise, you could be pouring perfectly good link juice down the drain.

When you want to exclude a page from the search engine indexes, do this:

<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
</head>

There’s No Stopping Bad Behavior

A problem you will have with both robots.txt and the robots <META> tag is that these instructions cannot enforce their directives. While Google and Bing will certainly respect your instructions, someone using Screaming Frog, Xenu, or their own custom site crawler can simply ignore disallow and noindex directives.

The only real security is to lock private content behind a login. If your business is in a competitive space, it will get crawled from time to time and there are few things you can do to stop or impede it.

One last note, I am not letting any cats out of the bag here. Pirates and hackers know all of this. They have known for years. Now you do, too.

• Conforming to search engine behavior
• Keeping-up with changes in search engine behavior
• Playing well with other websites to protect your SEO
• Avoiding common practices that obstruct SEO

Here are nine examples of what I call SEO quirks. See how many you know about.

1. In Subfolder & File Names, Use Dashes, Not Underscores

• Good: http://www.domain.com/sub-folder/file-name.htm
• Bad: http://www.domain.com/sub_folder/file_name.htm

Many developers favor separating words in file names with underscores (_) instead of hyphens (-). They are conditioned to this behavior because some programming languages reserve the hyphen, for example, as the subtraction infix operator.

On the other hand, Google was written for nerds by nerds; the search engine tends to see underscores as concatenation, or joiners, so technical terms like FTP_BINARY will appear on search results pages.

• If you are creating a new website, use hyphens.
• If you have a small website without a lot of inboud links, change existing URLs to hyphens and 301 redirect old URLs to new URLs.
• If you have an enterprise site that uses underscores, keep your old URLs and CMS rules, but switch from underscores to hyphens as the naming convention for all new file names.

A word of caution, keep the number of words and hyphens to a reasonable amount. On category name or topic level pages, I suggest short and sweet, one or two hyphens. When you create file names for articles, you have a lot more leeway; try not to go to town or stuff keywords.

2.  Avoid Dashes In Domain Names

Select a domain that is your brand or represents your business in a concise, professional manner without dashes. Do not worry about keywords. While I do not know of any technical reason not to use dashes in domain names, from a practical perspective, they look cheap and compromising. That may raise a caution flag when you reach out for links and citations.

One of the primary reasons people select hyphenated domains is to insert keywords. Last year Google updated its algorithms to dampen the exact match domain benefit. However, long before this, the success of numerous brand name domains, many verging on the ludicrous, proved you do not need a keyword rich domain to succeed.

3.  In Subfolder & File Names, Use Only Lower Case Letters, Numbers & Hyphens

Google and Bing are both terrific at handing complex URLS with spaces and non-standard or encoded characters. Where the problem lies is when other websites link to your documents. If you do not encode special characters, the content management systems of those websites that link to your documents may encode them.

For example, spaces become %20. If those websites use different character sets than what your site uses, they may not translate special characters correctly. The safest thing to do is keep it simple by using only a to z and 0 to 9 and hyphens.

According to the technical standard, URLs are case sensitive. Most content management systems handle mixed case addresses by rewriting them to lower case, but check yours and do not assume this.

Also, some analytics and SEO tools are case sensitive and will report different versions of the same URLs separately. The safest path is to make sure all your internal links are lower case and make lower case the style standard for all copywriters and coders.

4.  The Great Subfolders Vs. Subdomains Debate

It used to be that search engines treated subdomains somewhat like different websites. Today, they are roughly equivalent. In fact, it has been this way for some time. This is good because most third-party applications, like hosted shopping carts, must be in a subdomain.

Search engines are pretty proficient at telling whether subdomains are related or not. For example, Tumblr, Blogspot, and WordPress.com subdomains are not related, while www.domain.com and store.domain.com are related.

If you use subdomains, do not isolate them. Make certain the navigation links between your primary domain and subdomains are well integrated. I have seen applications inside subdomains that will only link to the primary website’s homepage or employ nofollow links.

5.  Be Careful With Parameters

Parameters are variables in URLs. The standard method for creating parameters is to end the address with a question mark, then list parameter names and values.

For example:

http://www.domain.com?category=billiards&game=nineball&article=how+to+rack+the+balls

Your content management system may rewrite this into a user- and SEO-friendly format,

http://www.domain.com/billiards/nineball/how-to-rack-the-balls

Both of the above URLs are fine. I prefer the second example as it is easy to read and removes unnecessary words and characters.

You definitely want to avoid missing or non-standard delimiters,

• http://www.domain.com?nineball (no parameters)
• http://www.domain.com,billiards,nineball (non-standard parameters)

I have seen some wacky delimiter schemes.

Be careful of user identification parameters like uid=142536 where each visitor gets a different number or tracking parameters like source=xyz where xyz is different for each referring document. These create duplicate content issues.

Your choices include:

• get rid of them; use cookies instead
• use the rel=”canonical” tag to tell search engines which URL to index and credit with links and citations
• tell search engines to ignore the parameters using webmaster tools (Google, Bing)

Another trick is to put parameters that do not affect page content after a #. Search engines almost always ignore everything after the # character in URLs, the exception being the AJAX hash bang.

6. Use Flash Or Silverlight To Insert Multimedia Elements, Not For All Content

While search engines tout their ability to crawl Flash and other rich media, they still do a poor job of it. Flash is great for inserting multimedia — like a video, animation, presentation, or sound file — into an HTML page.

Do not use an all Flash or all Silverlight website. Flash sites are particularly popular among artists, musicians, and photographers, which is a shame because these are people who could benefit from organic search. Keep in mind, Apple’s iDevices do not support Flash, so its popularity is waning anyway.

7.  Pick Only One Per Page, HTTP: Or HTTPS:

Google does not care if you use http: or https: — not!

While Google welcomes both http: and https:, on a URL-by-URL basis, pick one and stick with it. Let us say you have a shopping cart with secure https: checkout. If your crawler-friendly catalog pages resolve to both http: and https: versions, you could be in for a world of trouble.

I have seen websites where all the offsite links go to http: addresses and Google indexes the http: URLS. Then, all of a sudden, the addresses in the Google index change to https: for no apparent reason, and the website’s rankings disappear.

The simplest way to avoid this is with canonical tags that force http: or https:, whichever is the version you want indexed.

8.  Make Sure The Markup & Visible Text Matches

When the text in HTML markup does not match what users see, search engines call it cloaking. Sometimes, cloaking is unintentional. One example I saw occurred in a shopping cart where all the links to all the product categories and subcategories were included in the markup of every page. Visitors only saw links to the subcategories of the category they were viewing. The content management system hid the other subcategory links via CSS.

I do not want to get into a debate about white hat vs. black hat cloaking here, especially since Google engineers seem loathe to discuss specific cloaking techniques. Probably because they do not want to give people ideas. The two exceptions, ones they use for demonstration purposes, are serving different content based on user agents and using CSS to position text off the screen (-999 pixels).

They are always quick to say there are no good reasons for cloaking and that they have special detection algorithms that ring the red alert phone. The bottom line for this quirks article is avoid unintentional cloaking.

9.  Using The Vertical Bar In Title Tags

Search for long winding road, long – winding road, long — winding road, and long | winding road. Notice how Google ignores the dashes but not the vertical bar? That bar separates long and winding, not just visually, but in the Google algorithm. If your website uses the vertical bar, experiment by replacing it with a dash and see what happens.

If you knew all nine of my SEO quirks, good for you. It is not easy to maintain a current, comprehensive knowledge of all things Google and Bing SEO. Do you have any SEO quirks of your own? Share them in the comments.

I have to give special kudos to Paul Bruemmer’s Which Top SEO Tactics Will You Focus On In 2013? That is a tough act to follow. I don’t want to be redundant, so I am taking a different tact this year. While there is no way to make this wholly unique, here’s what’s on my mind.

Change How You Think About Search Engines

Search engine optimizers tend to think in terms of earning top rankings and placements. Perish any thought we’re manipulating Google or Bing, not to mention YouTube, App stores and directories, maps and review sites, or all those other search results out there. Yet, the cold truth persists, SEOs work to influence search engine results and we want them to reward our work by behaving a certain way.

This year something changed, something every SEO must confront during 2013. Search engines became both bold and adept at doing whatever they want to do… or don’t want to do. They will reduce organic screen space. They will change organic results to paid results. They will favor brand popularity over quality. They will reserve analytics data for paying advertisers. The list of changes (and complaints by SEOs) continues to grow.

Up to 2011, the test of a good SEO was wrangling search engines into behaving the way we wanted them to. During the last two years, it all ended. The vision search engines endorse, and their ability to enforce and execute that vision, grew dramatically closer. And, while the search engines’ algorithms evolved, so did their internal analytics, which means they can make more data-driven decisions.

Google and Bing are still far from perfection, and we will continue to see their mistakes, especially as they roll out new features and products. But, they aren’t pushovers anymore, not even for the most talented search optimizers.

Not being pushovers does not make them bullies. Get frustrated. Challenge questionable changes. Fight for more transparency and data access. Become an activist. But, don’t condemn the search engines. Without them, we would not have such tremendous opportunities. Understand the search engines’ goals, then use that knowledge to achieve your own business goals. Symbiosis is the mindset to adopt for 2013.

Go Enterprise

When we think of enterprise SEO, large sites with thousands of pages come to mind. Even if your site has only a few dozen or a few hundred pages, you need to embrace an enterprise approach. Large websites have long development cycles. They have to make smart decisions upfront because they have to live with them for a long time.

Get your navigation, internal linking, site performance, and page templates set-up and optimized now, then forget about them. Optimize the content you already have, then move on. If you’re constantly tweaking or changing these things, ask yourself whether you are creating impactful change or are frustrated because nothing else seems to be working?

Focus on new content. Create it. Evangelize it. Teach. If you’re not writing and publishing content yourself, provide topic and keyword guidance to those who do. Train your copywriters to optimize as they compose. Arrange to SEO review everything before it gets published.

This goes hand in hand with my next point.

Become An Online Brand

Search engines make it easy for major brands to rank well across lots of queries. They seem to give Fortune 1,000 and Global 2,000 companies leeway and forgiveness that small and mid-sized businesses do not enjoy. Right? Then, why do hundreds of lesser companies seem to enjoy the brand bump? Because they dominate their niches with great content and communication.

Content is king. This was the mantra at the beginning of SEO, and it is still true today. The difference is your content must perform triple duty.

• Keyword targeted and optimized
• Link worthy
• Social media worthy

Adding frequent, fresh content creates additional opportunities to rank well, but only if that content enjoys the necessary search engine authority, from either direct offsite links or internal links. This doesn’t mean everything you publish has to be link or social worthy, but you need successes and the occasional home run. It’s better to split time between creating content and promoting it than to keep churning our page after page.

Keyword Targeted & Optimized Pages

When it comes to SEO, if a page is not optimized for queries people search for, even the longest of long tails, it really isn’t optimized. Use keyword-research-generated ideas for fresh content and run keyword research on your ideas. It can go both ways.

Keep in mind, not every single page has to be keyword targeted. I contend you can always find one or two; but, if you have a great piece of content that you cannot match to any keyword, publish it and promote it. Every link adds to domain strength.

Link Worthy Content

Strive to create content that is so awesome other websites will be compelled to link to it. Make it interesting. Make it high quality. Create linkbait.

Did I write create linkbait? Linkbait has become an ugly term. Don’t buy into it. Read what Google’s Matt Cutts says:

Linkbaiting sounds like a bad thing, but especially if it’s interesting information or fun, it doesn’t have to have negative connotations. I hereby claim that content can be both white-hat and yet still be wonderful “bait” for links.

Matt Cutts – http://www.mattcutts.com/blog/seo-advice-linkbait-and-linkbaiting/

Linkbait strategies are approaches to writing content that’s psychologically primed to attract links. Let’s trash the negative connotations and bring back linkbait in 2013. If you want to learn more about linkbait, read The Link Baiting Playbook: Hooks Revisited by Todd Malicoat.

Social Media Worthy Content

Can you summarize your content in 100 characters plus a shortened link and make it compelling enough for people to click? In a nutshell, this is what social-media-worthy means. Please don’t equate social-media-worthy with popular. If George Takei and I made the same post on Facebook at the same time, I might get a dozen likes while Mr. Takei would get anywhere from 10,000 to 80,000 likes. Creating social media worthy content and posts is only the first part.

Social media for SEO must be accompanied by audience development, especially when you don’t have enough followers to create a critical mass of activity. This is where influencer marketing, community building and lots of other social media buzzwords come in.

I’ll refrain from explaining the flywheel analogy and just return to what I wrote earlier; take time to promote the content you create. Also, take time to participate, build relationships, promote others, and reciprocate. If you are doing social media for SEO, this is a good place to spend a chunk of your time.

Social Media For SEO

Do not confuse social media for SEO with social media for conversions. They can overlap, yes, but often they don’t. You want to identify and foster an audience that will follow, like, share, retweet, etc. If the people who make buying decisions do not congregate together online, you need to find an audience that does, one you can engage. This is often the case in B2B. If your social media program does not create search engine ranking authority, you need a separate social media for SEO campaign.

The Third Screen

There are three types of screens: desktops, pads, and mobile. Most people use pads as second screens inside their homes or offices. Smart phones are largely used outside of the home and office when a larger screen is not handy. You can be sure the search engines are tracking what types of results users select on these different devices. As they get more and more data, I expect the results to become increasingly customized by device type. Hopefully webmaster tools will add pads as a device type this year.

Structured Data

Google’s new WYSIWYG tool for identifying structured data is an interesting development. As sites use this, it will provide the search engine with a wealth of new data it can apply to create or tweak algorithms that identify “structure-able” data on any website, and then influence both Google’s search results and presentation. I imagine this type of application is a ways off, but it does signal Google’s interest in identifying and using structured data. If you haven’t yet, add RDFa or Schema.org markup into your pages and templates.

Reputation & Trust

Two words I frequently saw or heard this year were reputation and trust. I am writing about the credibility of a website’s content, design, and external links, not reputation management as an SEO service.

Since day one, Google and its fellow search engines promoted consistent quality while criticizing trickeries. What changed is Google can police bad behavior more effectively and broadly than ever before. Google got teeth.

Google Gets Vocal

In the past, Google shied away from notifying domains about their black hat webspam via Webmaster Tools. This changed in April when the search engine expanded the types of messages and warnings it sends.

Also see Google Sent Over 700,000 Messages Via Webmaster Tools In Past Two Months.

Penguin

Google introduced Penguin on April 24th. Penguin penalizes websites that exhibit signs of artificial external links. This is what Penguin looks like in analytics.

When it comes to recovery, Google seems adamant that websites must make a thorough effort to remove all artificial and low quality links no matter how old. The search engine did provide a link disavow tool last month, but it uses submissions as strong suggestions, not a hard and fast off-switch.

Even with the disavow tool, Google makes no rush to restore a site’s good status. Google waits until it recrawls and reindexes the URLs you disavow before it takes action. It can be weeks or months between spider visits to deep or low quality pages. Finally, I have seen no change in Google’s statement that some domains are beyond rehabilitation.

It is worth noting here, Google ignores links it cannot trust. It seems websites can have plenty of unreliable links before they pass some statistical threshold and Penguin takes hold. Penguin does not replace manual reviews, either. Google may still take manual action against a site because of untrustworthy links, even if Penguin has engaged.

Panda

Google loves Panda, the algorithm that penalizes websites for too much low-quality content. Since November 18, 2011, Google updated Panda 13 times. Panda acts like a ratio-based punishment. The sites I have seen recover remove inferior content and replace it with well-written, useful pieces. They also combine or better differentiate duplicate or near-duplicate content. A good example is the company that puts up a separate page for each office location, the text on all pages being identical except for the city, state, and address.

Rewarding Quality

It often feels like Google spends all its time looking for low quality, so it was nice to see changes to identify and reward high quality in the search engine’s June-July weather update.

Webspam

Back in April, Matt Cutts announced this news:

In the next few days, we’re launching an important algorithm change targeted at webspam. The change will decrease rankings for sites that we believe are violating Google’s existing quality guidelines.

http://goo.gl/LnG1l

Forecasted to affect 3.1% of queries, Matt was vague about how the algorithm works. One example showed blatant keyword stuffing. The second example showed links in spun content. Because Matt wrote:

Sites affected by this change might not be easily recognizable as spamming without deep analysis or expertise, but the common thread is that these sites are doing much more than white hat SEO; we believe they are engaging in webspam tactics to manipulate search engine rankings.

I suspect the update includes some form of language analysis.

Over-Optimization

Last March, Matt Cutts announced an upcoming over-optimization update.

We are trying to make GoogleBot smarter, make our relevance better, and we are also looking for those who abuse it, like too many keywords on a page, or exchange way too many links or go well beyond what you normally expect.

What is in the over-optimization penalty? We do not know, but the SEO community has ideas. When he made the announcement, Matt mentioned too many keywords on a page, something he has described before.

This month, Matt spoke about site-wide backlinks and compared how Google counts these to how Google counts keywords. I suspect site-wide links are part of the over-optimization algorithm.

In the image below, I illustrate a generic example. One instance is good, two is better, three or four is great, and then each additional mention becomes less and less important until you over-optimize. At some point, your optimization becomes suspect.

To be clear, I picked the Golden-Ratio and number of instances arbitrarily in order to convey this concept. The real Google formulas for stuff like keyword frequency and repetitive links are unknown. The number of instances will vary too. The point is, do not try to outguess Google. Be natural.

What I find especially interesting is that Google created a safety net for some things. If you have legitimate site-wide links, like a blog roll or links to subsidiary companies, Google will not penalize your domain. Also, and I am reading deeply between the lines on this one, it sounds like some things may not trigger an algorithmic penalty, but could be disastrous during a manual review.

Exact Match Domains

In September, Google announced they would crack down on low quality exact match domains. While this is not related to Panda and Penguin, it does target exact match domains that rank well because of their TLD and not their content or external links.

Too Many Ads Above The Fold

Sites that have too many static advertisements above the fold and force readers to scroll down the page to see content risk incurring a penalty. This does not affect too many sites, Google says less than 1%; so, it clearly targets outliers.

Infographic & Guest Blogging Links

We do not know of any actual update, but in July, Matt Cutts warned that infographic links are getting abused and may become a target of the webspam team in the same way widgets were discounted and penalized.

In October, Matt Cutts offered a similar warning to blogs and guest bloggers. White hat guest blogging can be a terrific win-win, but shady guest blogging may have consequences.

Pirate Update

I will finish my examples of reputation and trust with the Pirate Update. It is a penalty against domains that receive too many DMCA “takedown” requests. This one appears to be a straight-forward tie-in between Google’s webspam algorithms and its DMCA request database. There are some important exceptions, so check out the link.

Google on Caffeine

It has been a couple years since the Caffeine infrastructure rolled out. Last year, we got a deep-roasted taste of Caffeine, thanks to Panda. This year, the Penguin, site-wide links, and ads above the fold algorithms appear to take advantage of Caffeine, too, probably in collaboration with increased crawling, data storage, and processing capacity.

Freshness

A year ago, Google launched its freshness update, affecting 35% of search results. This much improved ‘query deserves freshness’ algorithm identifies recent or recurring events, hot topics, and queries for which the best information changes frequently.

AJAX & JavaScript Execution by Spider

Another example of Google’s new abilities was the announcement that its search spiders can now execute AJAX and JavaScript. While the announcement appears targeted toward reading dynamically generated comments, this capability has wide implications. Previously, it was common practice to hide links behind JavaScript, a form of old school cloaking or PageRank sculpting. This will not work if Google can read those links.

Indexing iFrame Content

Michael Martinez designed a test demonstrating how a link on the iFramed page passed a unique anchor expression to another page. This did not work on Bing, and I am definitely not endorsing iFrames. iFrames were once widely used to hide content from search engines so it is a worthwhile demonstration of Google’s expanding powers.

Automatic URL Canonicalization

Maile Ohye spoke about this at SMX Advanced, and it caught my ear then. However, I did not think much about this until I saw Matt Cutts’s latest video. Duplicate content and canonicalization has always been a cornerstone of SEO.

Now Google says they will detect and group duplicate content URLs then combine their authority. While it is still important to do URL canonicalization via rel= tags or in Webmaster Tools, dynamic de-duping and combining authority from multiple pages is a noteworthy innovation.

Parked Domains and Scraper Sites

Last December, Google added a parked domain classifier to keep parked sites out of the results. They also improved the ability to detect duplicate content and show the originating document in search results. Removing parked domains may not seem like a big tech leap; it does demonstrate Google’s capacity growth. It’s the same with scraper sites. All that data has to get stored and cross-referenced.

Domain Diversity In Results

In September, Google released an update to increase the number of domains that appear in search results. As Danny wrote, “ Google’s search results can sometimes be dominated by pages that all come from the same domain.” This update is supposed to help alleviate this. That adds another layer of processing to the rankings selections.

Tags & SERP Real Estate

Before I finish, I want to discuss two more things: tags and real estate. Both continue to evolve and both are increasingly controversial.

Tags

Google is pushing tags, and you need to keep up on them. Tags come in two broad varieties, machine-readable mark-up and HTML elements or attributes. You must absolutely understand how to use HTML tags like rel=canonical and rel=author. Learn how to use these and incorporate them. Push your CMS developers until they support them.

Whether or not you use machine-readable markup like Schema.org is another matter. Yes, they make it easier for search engines to discover, classify, and display information.

They also make it possible for search engines to pull information from your website and display it in the search results, possibly costing you visitors and traffic. Whether or not this really does divert traffic is being hotly debated. What is important is to become informed and make the right decision for your business.

SERP Real Estate

In the last year, Google made more changes to how they display search engine results than ever before. The latest examples are killing the left sidebar and eliminating non-paid product search. More queries trigger local search results, a benefit to some businesses, while a detriment to others. Some queries only deliver seven organic results, no longer ten. Increased site links. More above the fold advertisements. Who knows what is to come?

I think Google wants to simplify the search results for the average user while maximizing its income opportunities. The typical user is not going to miss the power tools on the left side or look for the new dropdown menus. Google wants to personalize the results, too, with more emphasis on local and friends.

Search for coffee. In my own Google results, I see seven local results, three with personalized recommendations. Below that are three more personalized results in the organic listings.

This is where Google SEO is headed, personalized social and local results. More on that in my column next month.

Actually, I would not ask you this question without reason. Being an inbound marketing consultant, I get to enjoy conversations with people in many different roles.

From the c-suite to marketing and sales directors, to Web designers, developers, copywriters, community managers, social media managers, and search engine optimization professionals — I get to talk to them all.

SEO Is Scary!

Each of these people has a personal point-of-view shaped by their roles and responsibilities. When you begin talking about search engine optimization, some of them get downright hostile. It is understandable.

They are knowledgeable, work hard and become wary when others, especially outsiders, want to change how they perform their jobs.

As an example, I had a conversation with a social media manager. She sees her role as building brand awareness, enhancing her employer’s reputation, and driving traffic to the company website. She writes about company events and products. Talk to her about community and traffic, it is all smiles and excitement.

However, when you mention using social media to drive SEO, she withdraws. Why? Because writing about webinars, white papers, newsletters, product announcements, company news, and whatever else comes down the pipeline gives her a wealth of things to discuss.

More importantly, her enjoyment and excitement are genuine. To her, search engine optimization means having to contrive posts that feel forced, fake, and unnatural.

Here is another example, one I know many SEO consultants have faced. How many people search for products or services? Then why is the navigation for so many websites:

• Home
• Products
• Services
• Support
• About

Add other frequently seen top navigation links like resources or solutions or press, and you have the makings for every cookie-cutter business website. These sites pick two or three top keywords, stick them into their homepage title tag along with the business name. They pile all the major keywords into the title of their product or services page.

Then, they use the keywords again for each individual product or service. Before you know it, these websites have two or three pages competing for the exact same keywords, and no one can give the copywriters clear instructions about which keywords should link to which pages.

Should you bring up the idea of rebuilding the site navigation around the business’s products and services, you might find yourself cornered by a defensive website designer explaining how this will destroy the site’s logical order, or that the content management system will not allow this type of navigation.

How do you, the search engine optimizer, circumvent this type of pushback? How do you take the scary out of SEO?

Stop Thinking Of SEO As A Marketing Channel

Search engine optimization is a set of practices and principles to apply to your marketing channels, not the other way around. Yes, non-paid search is an important source of traffic, but organic search referrals come to your website as a result of what you do on your website and social media accounts, and as a result of the authority people and websites bestow in the form of links and brand mentions.

You can influence Google, Bing, and other search engines, but you cannot place messages on them directly like you do on your website, social media accounts, press releases, or advertisements. As an analogy, Forbes, Fast Company, or The Wall Street Journal might write an article about your company, but you would not call this a marketing channel.

Evangelize SEO As A Company Goal

Get search engine optimization out of the marketing department and make it a company-wide responsibility. The proper place for a company’s SEO mandate is from the chief operating officer. While the actual training and daily evangelizing will likely come from the marketing department, SEO extends beyond any one person or section.

The website development team must implement on-site optimization of HTML markup. Your sales team is responsible for income. Finance is responsible for measuring and reporting ROI. While roles and responsibilities vary from company to company, compartmentalizing SEO will lead to pushback.

SEO extends offline. We know social media success affects SEO. Do your business cards carry the company Twitter address? If employees speak publicly or make sales presentations, do they put the Twitter address on a slide and invite attendees to follow? Look for avenues of indirect support that can help your SEO efforts.

Find ways for people to contribute to SEO, either directly or indirectly.

Make SEO A Responsibility

Not everyone can be responsible for SEO, either as a whole or for even a small part. But, as I have shown, many can. For these personnel, include how they can contribute to SEO in their job descriptions or work plans and make it part of their performance reviews.

Provide Education & Training

When you make SEO an employee responsibility, you do not want to shove it down everyone’s throats. Critical to earning buy-in from your personnel is education and training.

At the very least, you want employees to understand how the company operates and generates income. Just as more and more businesses are developing social media policies and training, search engine marketing should be part of their education.

For those working closest with SEO — your social media managers, copywriters, Web designers, and others — provide them with formal search engine optimization training, then include them in planning and discussions.

As a consultant, I do not want my first meeting with your website designer to be about reorganizing the navigation he or she spent many hours of time and effort crafting. I do not want to walk into your sales department, cold, to talk about SEOing their landing pages, especially after they spent months of performance testing and perfecting.

Change is inevitable. So is pushback. By educating everyone first, and asking them to think about how to enhance SEO, conversations will be more participatory and productive.

Good SEO Is Planned SEO

Your sales team knows what new offerings are coming for weeks or months in advance. This lets them plan, create materials, and schedule activities. Likewise, search engine optimization should use foreknowledge to plan SEO strategies, tactics, and activities. Minimize on-the-fly decision making as much as is practical.

Some decisions must be centralized. For example, one person should be assigned as the keeper of the keywords, given the responsibility for deciding which search queries will become targets and on what pages. You may not want one person targeting a blog article while another person targets a product page, both for the same keyword.

Identify potential points of conflict or failure and put somebody in charge. This is a terrific role for a dedicated in-house SEO professional who can work with different teams and departments.

Provide Specific Information To Those Responsible

If you want to drive people crazy, tell them to optimize for a keyword without giving them resources or guidance. Alternatively, ask someone to optimize a page without providing any keywords. These types of things occur all the time.

Provide keyword and target URL lists, then make sure everyone knows how to use them. Also give SEO team members access to analytics, webmaster tools, research tools and other resources that they will need to make smart decisions.

Search engine optimization does not happen in a vacuum. It is not a solo effort. Not every site change, blog article, or social media post needs a full-blown campaign. But when they do, personnel need to know who to go to for supporting articles, internal links, page inserts, press releases, and other parts of a campaign.

Create a clear pathway for choosing the proper level of planning and to create successful campaigns.

Hold SEO Team Meetings

In the enterprise workplace, the decentralization of SEO is inevitable. Get all the major players together on a regular basis. I suggest weekly or twice a month, depending on company size and how much organic search traffic contributes to ROI.

Include Web design, social media, copywriting, and anyone who has SEO as an important responsibility. Who leads these meeting can vary from company to company; usually it can be the marketing director or head of online marketing. He or she should be senior enough to make and enforce decisions. Include representatives from paid search, online advertising, and affiliate management.

Topics to cover include a rundown of SEO analytics, social media analytics, SEO industry news, company news (upcoming announcements, new products and updates, company events, events employees will attend/speak, etc.). Have each attendee make a report so everyone knows what is happening among the departments. Before the meeting, each member should give the chair a list of things they need input on or that require a consensus decision.

Over To You

SEO can have multiple points of both opportunity and failure, especially in large enterprises. What do you do to make sure you optimize your search engine optimization planning and execution? How do you take the scary out of SEO? Comments, if you please.

Image: Pennywise the Clown from the movie IT

Some content management systems bake poor URL structures right into their websites. Lax rules can be a culprit, for example, not encoding spaces or special characters.

Meanwhile, some CMS platforms devise URLs using illegal characters that should not appear in addresses. Others generate multiple URLs for pages, creating duplicate content.

While it is true that search engines go to great lengths to read and index even the worst URLs, attention to URL management and optimization will provide both SEO and usability advantages.

Good URL Structure

A few years ago, Dr. Peter J. Meyers put together a cheat sheet on the anatomy of a URL. It’s a good one to keep handy.

Look at this URL:  http://www.mobilesmart.com/phones/android/samsung-galaxy-s3

• It is easy to read and understand. If I saw this address pasted into a blog or forum, I would likely click on it.
• It is SEO optimized with breadcrumb style keywords. Search engines look for keywords in URLs; it’s a known ranking factor. This layout, going from general to specific, is ideal for enterprise SEO.
• The URL includes its own anchor text. If this address were pasted into a blog or other web page as a link, that link would possess well-optimized anchor text.

Old style dynamic addresses are legal and acceptable, though they have drawbacks.

• They tend to be longer and difficult to read because they contain both parameter names plus values.
• Pairing parameter names with values adds extra words. This may dilute the SEO value derived from keywords within the URLs.
• This type of address may contain information better transmitted outside of the URL. A user ID, session ID, sort code, print code and many other possible parameters could create duplicate content, security or other issues.

Diagnosing URL Issues

To find URL based issues:

1. Check for errors and warnings then determine if URLs are the culprit.
2. Audit all URLs for proper syntax.

To check for errors, begin with Google and Bing webmaster tool reports. Look for duplicate content then examine the webpage addresses themselves and their locations. Numerous third-party SEO tools can locate SEO issues as well.

Canonical issues, parameters that do not change page content, loose adherence to coding standards, or any number of reasons will create duplicate content.

Options for dealing with duplicate content include:

• Reconfigure the content management platform to generate one consistent URL for each page of content.
• 301 redirect duplicate URLs to the correct version.
• Add canonical tags to webpages that direct search engines to group duplicate content and combine their ranking signals.
• Configure URL parameters in webmaster tools and direct search engines to ignore any parameters that cause duplicate content.

I worked with a newspaper that used unique numerical identifiers, outside of parameters, to serve articles as webpages. It did not matter what the URL contained, as long as the identifier was somewhere in the address. Unfortunately, the writing of link hooks into templates was inconsistent, resulting in thousands upon thousands of duplicate content pages. We had to pour through each template, rewrite each link hook as an SEO friendly URL, then catalog all the legacy URLs and 301-redirect them to the new optimized addresses.

When auditing URL syntax, I prefer to export every webpage address into a spreadsheet or database. If you’re thinking about using Google site: queries, don’t bother as many of the issues you will look for do not appear in search results.

Search For Reserved & Unsafe Characters

Reserved Characters

Each character has a specific use. Should they appear, determine if they are used properly, should be encoded, or if the URL needs reconfiguration.

Unsafe Characters

Encode unsafe characters unless used for a specific purpose. The % symbol does not require encoding when used to encode a character. The # symbol does not require encoding when used to create an anchor tag.

Miscellaneous Characters

Strictly speaking, these characters do not require encoding. In reality, many CMS platforms will encode these automatically. If you want links that contain these characters to remain consistent when shared from website to website, it’s a safe bet to encode these.

Search For The Pound Symbol, #

Search engines ignore the # and everything after it in a URL. If using the #, make sure the webpage appears as you want it crawled and indexed when the # and everything that follows is removed. If the # changes content you want indexed, you will need to find a different URL structure. For example,

• /celebrities.html#bill-clinton
• /celebrities.html#bette-davis
• /celebrities.html#deadmau5

Based on these webpage addresses, let’s assume the webpages are all different. This will be a problem because search engines will index only /celebrities.html.

A better URL would be /celebrities/deadmau5 or /celebrities/bill-clinton.

Search For Underscores, _

Underscores, while legal, are problematic for SEO. It’s an issue search engines have always dealt with but never solved. Search engines see underscores as connectors. To separate words, use dashes.

So, in practical terms, while hello-dolly is hello dolly, hello_dolly is hello_dolly.

Always uses dashes, -, to separate words.

Search For Mixed Case

URLs, in general, are case-sensitive (with the exception of machine names). Mixed case URLs can be a source of duplicate content. These are not the same URLs,

• http://example.com/Hello-Dolly
• http://example.com/hello-dolly

The easiest way to deal with mixed case URLs is to have your website automatically rewrite all URLs to lower case. With this one change, you never have to worry if the search engines are dealing with it automatically or not.

Another great reason to rewrite all URLs to lower case is it will simplify any case sensitive SEO and analytics reports. That alone is pure gold.

Check Your CMS Platform’s Settings

A major clothing retailer I worked with uses a popular retail CMS for enterprise. When this client came onboard, it had some of the nastiest URLs I’d ever seen. I wanted to blame the CMS, except other retailers using the same platform had gorgeous webpage addresses.

In our CMS audit, we found that the client left the optimized URL field blank, leaving the CMS to default to non-optimized, sketchy addresses.

Creating Optimized URLs

How you get from your present URL structure to an SEO optimized one depends on your content management system.

• In WordPress the administrator select the permalink structure, define and create category slugs. Writers can edit the slug that will become an article or page URL (if the slug is part of the permalink structure).
• In some CMS programs you can create almost any URL structure you want by placing links into templates. You just include the correct page identifier and pull from accompanying variables (ex. http://domain.com/{article_category}/{article_id}/{article_slug}).
• Your developer may have to create dynamic URL rewrites.

To find the best path for your business, speak with your Web developer or CMS vendor.

Meet The Minimum Syntax Standard

Not all websites can optimize their URLs; some CMS platforms will not support it. However, no website should use illegal URLs. If you cannot optimize, make certain that you at least meet the standard.

Further Reading

• http://en.wikipedia.org/wiki/Uniform_resource_locator
• http://tools.ietf.org/html/rfc3986
• http://www.w3.org/Addressing/URL/uri-spec.html
• http://www.w3.org/TR/WD-html40-970708/htmlweb.html

Image credit: SEOmoz Anatomy of a URL

If your domain and homepage are different, determine how the domains forward visitors to the actual homepage. The correct way to forward visitors from one Web address to another is via a 301 Permanent Redirect.

A permanent redirect tells search engines to take the authority from off-site links pointing at your domain and apply it to your real homepage.

The most commonly used incorrect way to forward your domain to your homepage is a 302 Temporary Redirect. A temporary redirect tells search engines not to forward any authority from links. It’s like burning money. The authority evaporates.

On many websites, the majority of off-site links point to the root domain, with or without a preceding www.

For example, www.akaipro.com 302 redirects to www.akaipro.com/en/index.php.

www.akaipro.com

Open Site Explorer Page Authority: 76 Followed Linking Root Domains: 858 External Followed Links: 20,479

www.akaipro.com/en/index.php

Open Site Explorer Page Authority: 68 Followed Linking Root Domains: 231 External Followed Links: 2,828

This domain is not using 88% of its SEO friendly inbound links for homepage SEO authority!

What’s Happening? What Should You Do?

Bloggers and other online content creators like linking to brands. When not linking to specific content, they often link to the root domain as a default. It is in your best interest to make use of the search engine ranking authority these off-site links send your way.

Having A Different Domain Than Homepage

Having a different domain and homepage is not automatically a bad thing and there can be a good reason to do so.

For example, your website might forward visitors to the correct language or country version of your website based on their location. You just need to configure things so as not to block incoming link authority and within search engine terms of service.

Let’s look at another company, Denon.

• The root domain is denon.com
• Websites link to www.denon.com
• The global homepage is www.denon.com/Pages/home.aspx
• The USA homepage is usa.denon.com/us/pages/home.aspx

Denon 302 redirects visitors from the root domain to the global homepage. If you visit the USA page, the website sets a cookie. As long as the cookie is set, the site will:

• 302 redirect from www.denon.com to www.denon.com/Pages/home.aspx
• 302 redirect from the global homepage to usa.denon.com
• 301 redirect from usa.denon.com to Location: usa.denon.com/us/pages/home.aspx

Denon fails the 301 redirect test from www.denon.com to www.denon.com/Pages/home.aspx but they get it right redirecting usa.denon.com to usa.denon.com/us/pages/home.aspx with a 301.

But what if Denon automated which homepage it served based on my IP address location? The proper procedure would be to:

• 301 redirect the root domain to the global directory page
• Set a cookie
• Test for the cookie
• If the cookie is there, 302 redirect visitors to the appropriate custom homepage based on their IP address or another means of location
• If the cookie is not there, serve the global directory page

The key to any automation is to treat people and search engine spiders the same. Search engine spiders do not store cookies, normally. You treat the search engines exactly the same way you treat any visitor who has cookies disabled. It may seem like splitting technical hairs, but it is important to stay within the search engines’ terms of service.

Whatever you do, do not sniff for search engine user agents or search engine associated IP addresses. If a cookie-enabled search spider happens to visit your site, redirect it to the correct homepage based on its IP address. Rand Fishkin wrote a great article about this.

Search engines are doing some crawls with cookie enabled spiders, but I’m confident this passes the smell test and would pass a manual review because it is neither cloaking or deceitful.

Best Practices When Domains & Homepages Are Not The Same

Ultimately, you should be 301 redirecting all versions of your domain to the master homepage.

• http://exampledomain.com (non-www version)
• http://www.exampledomain.com (www version)
• http://www.ExampleDomain.com (mixed case versions)

Do not redirect everything to the www version of your root domain then redirect to the master homepage. One of the websites I polled 301 redirects all domain versions to www.domain.com then 301 redirects that to www.domain.com/homepage.

Every time you 301 redirect, a little link authority is lost. To conserve as much authority as possible, make certain there is always just one redirect hop and that it is a 301 redirect. It’s worth knowing too that search engines will likely stop following redirects after four or five hops.

If you want to forward people to a custom homepage, for legitimate reasons, do it from your master homepage only, use a cookie test and then redirect them.

How You Can Tell?

You can locate all redirects on your website by running your own crawl with tools like Screaming Frog. These will show you where the redirects are, where they go and whether they are a 301 or 302.

If you want to test one address, such as your root domain, I set-up a simple Web Page Server Response Code Test that will tell you if your domain and homepage are the same and, if not, what type of redirect your website uses.

Links are always a popular topic, and this year has proven no exception including Penguin, links vs. social media, negative SEO and disavowing links. Recently, someone asked me to distinguish between good links and bad links.

First, there are really three types of links:

1. Links that help SEO
2. Links that hurt SEO
3. Links search engines ignore

Google and Bing have been clear, search engines do not use every link as a ranking signal; they ignore many links.

While they are not going to tell webmasters which specific links matter and which ones they ignore, the search engines do share generous amounts of information concerning what types of links they deem beneficial, manipulative or ineffective.

We can also apply our understanding of how search engines operate to apply a little deductive reasoning. Pretty much anything you can quantify, you can place on a distribution curve.

Search engines can apply statistics to the link data and other SEO signals they collect to tell which websites are within a normal range and which websites exceed or lag behind what is natural for a particular ranking factor or group of signals.

For example, if the mean percentage of phrase-match anchor text for wedding jewelry is 17%, then a document with 5% phrase-match anchor text lacks relevance and a page with 30% phrase-match anchor text exceeds the norm to the point it’s probably over-optimized or manipulative.

• With positive SEO signals, a site should be above average, but not to the point of exceeding reality.
• For negative SEO signals, anything from zero through normal is likely okay, but exceeding what is typical may hurt your rankings.

I am not going to get into standard deviations, types of ratios or degrees of freedom. There are many different statistical tests search engines might choose from, the bell-curve being a simple example. One or two outliers likely will not mark a website as being manipulative either unless it is something egregious or that search engines are sensitive about.

Naturally occurring exceptions can be expected. It is more probable search engines look for suspicious trends across multiple statistical signals before exacting a penalty. Remember, I’m hypothesizing here.

As a practical example, a few paid links most likely will not hurt your rankings. They could be part of a legitimate advertising campaign. Google and Bing will likely ignore them.

At some number of paid links, though, the search engines will observe that you have exceeded what is usual or normal and apply a penalty. This could apply to reciprocal links, links in badges or any type of links they consider dodgy.

Another reason I believe search engines look for patterns across different types of signals is to prevent negative SEO. It would be easy to buy text links and point them to competitors. Creating a portfolio of many different types of negative links from a large variety of sites would be far more difficult to accomplish.

A Closer Look At Different Types Of Links

Anchor Text

Keywords in anchor text, the words in a link, are not a link type, though it is worth including. This is a signal of relevance. If enough sites link to your document about baseball with the word baseball, the search engines take it as a signal that other sites consider you page useful and relevant about baseball.

In the real world, people do not always use keyword specific anchor text. Often they will use your brand or company name. They may use the title of a page or article. They might use text that they believe readers will find useful or something generic like click here. Keywords in anchor text is a great thing to have.

Use strategies like including a keyword prominently in your H1 or article title. However, do not obsess. You want anchor text to vary. Too many links with your exact or phrase-matched keywords eventually becomes abnormal.

Reciprocal Links

Reciprocal links or link trading is one of the oldest SEO strategies around. It was one of the first types of links search engines warned us off, too. If anything, there is too much fear around reciprocal links. You should do what is right for your business and website. It makes sense for businesses that work together or share relationships to link to each other.

For example, in many communities, antique stores offer free maps of all the antique shops in town. It makes sense to recreate this map online and for stores to link to each other. What does not make sense is for an antique shop to link to a car dealership, especially if it is in another city, state or country.

Reciprocal links may not help with competitive keyword rankings, but that does not mean you should avoid them when they make sound business sense.

What you should definitely avoid are manipulative reciprocal linking schemes like automated link trading programs and three-way links or four-way links. This is when site A links to site B, site B links to site C and site C links to site A. I can think of no legitimate reason to do this and search engines see these patterns.

Directory Links

Directories are another greatly maligned source of links and with some justification. When you see a site named Free SEO Link Directory or a network of directories, “Pay $25 to get instant inclusion in 25 different directories with custom link text,” the red siren light should be spinning.

Search engines do their best to ignore these. These are a waste of time and money.

Should you be in directories? Yes, when they serve a legitimate purpose. If you are a lawyer, it makes sense to be included in lists of local attorneys. If you have a wedding business, you want to get included in popular bridal directories.

Always consider the source. Is it a professional organization? Will the visibility enhance your status? Will it send relevant traffic?

Links in Content or Blog Networks

Content networks were popular among some SEO sects for quite a while, until Google unceremoniously began taking away their ability to rank.

The practice included cheap articles and article spinning. Write a bunch of articles, upload them to a central management program and have them published on a bunch of different blogs or websites. Smart SEOs used content networks as part of advanced schemes like link wheels. Less savvy users linked content networks directly to pages they wanted to rank.

I can think of few reasonable uses for content networks. They are easy to uncover too. All you have to do is identify one blog or site then look for other websites that link to the same websites.

With a little effort, one can unravel the whole thing, so it should be no surprise search engines built an algorithm to accomplish this. Another downside to content networks is that once your content is published, you may have no way to take it down when Google sends you a suspicious link notification.

There are a few carefully curated, high quality content networks out there. If you have one or get invited, count yourself lucky.

Article Sites

Another old school link building tactic is publishing articles on free article sites. Write different versions of the same thing, put it on different article sites and get links. With luck, other websites will copy your articles and links onto their websites. Search engines stopped counting links from article sites long ago.

Further, copies of articles on other websites can be easily identified and the ability of those pages to pass PageRank or link authority turned off. This is another waste of time.

Press Release Links

While press releases are still popular and part of many SEO strategies, I believe they are over-hyped. Many experts feel that the search engines ignore links in press releases, on press release sites, because they amount to pay-per-post content. It probably does not help when blogs or scrapper sites pick-up and re-publish press releases either. Like article site content, press releases on additional websites are duplicate content.

The proper way to use press releases is as part of a true media relations campaign. When bloggers and journalists write original stories based on your press release or outreach efforts, the resulting links can be valuable. This is why PR , Social Media and SEO teams or departments must collaborate.

Blog Rolls

Blog rolls can be good for traffic and publicity, but they have limited SEO value. Most blog rolls appear in sidebars and on every page. Without knowing exactly how search engines count blog rolls, it is probably safe to consider these as one link per site. Chances are good that search engines count them, but greatly depreciate them.

Text Links

Avoid off-site text links outside of content, especially in sidebars and footers. Site wide text links are especially ominous. Imagine how having Colorado Fine Silverware link to your website from the footer of every page on numerous sites looks to search engines.

Legitimate text link ads do exist. Search engines try to ignore these, but probably have limited tolerance. Buy too many ads and it begins to look like rankings manipulation. If search engines identify your text link ads they will not help SEO, but buy too many and it could hurt.

Badges and Apps

Creating badges or small apps with links in them may seem like a clever way to build authority, but search engines found the links are usually more about SEO than relevance. Some prominent websites have lost their rankings when Google began ignoring badge or app links. This can be a good promotional tool, but I suggest adding NoFollow to any links.

Infographics

Recently, Google announced it may suppress infographics from passing PageRank. This is not surprising as most infographics are created more for link building by regurgitating content and information that’s already out there from original sources. If you rely on infographics heavily, start looking for new tactics.

Social Media

Social media is a great source for links. You need to consider the relationship each site has with search engines.

For example, Google+ is part of Google and links from people in your circles can appear in your personalized rankings and search results.

Bing has relationships with Facebook and Twitter. They prominently include links by your Facebook friends and Bing plans to increase the visibility of Twitter content.

Microsoft’s Aya Zook recently wrote to me about Twitter:

Yes, the Twitter partnership is ongoing. In the current Bing experience, there are two areas where Twitter results are prominently displayed.

• News: The results appears beneath the most recent news results on the right-hand side.  Try searching for a timely topic – “Katie Holmes” for example – and if you click the News vertical, you will see public tweets related to the topic.
• Sidebar > People Who Know: We identify experts or people that are enthusiasts or authoritative on various queried topics based on a) blogs, b) Twitter, and c) other social networks like Google+, Quora and LinkedIn. Our algorithm considers factors like the number of tweets and re-tweets, number of followers, blog readership, recency of shared content, etc. to help ensure people we surface are influential on a topic. After connecting Bing to your FB account, when you type in queries related to topics where experts might exist – e.g. “photography” or “wine” – authorities on the topic will surface under “People Who Know” in the Sidebar experience. As we keep expanding the topics that trigger the experience, more and more experts will be served from across a myriad of social channels beyond Twitter.

Yes, [Twitter] coverage is low still but should rapidly ramp up over the next few months.

Even when search engines do not have a relationship with social media sites, they will crawl content outside of the firewall, but their ability to crawl and index is limited. Most links on social media sites are NoFollow and do not pass traditional link authority.

Forum Posts

Links in forums can be tricky because spam is rampant within many communities. Consider the quality of the forums.

For example, a popular forum on a product manufacturer’s website will be far more likely to pass authority than some forum where everyone is self-promoting with links in most posts. Assume search engines will ignore links in signatures.

The key is to participate in forums to help others, not to add links. Forums can be a good place to build relationships with people who will naturally share your content and links.

EDU and GOV Links

It is worthwhile mentioning .edu and .gov links. While links on university and government sites continue to be highly prized, search engines give no special consideration given to these sites. They can pass high authority because they enjoy large numbers of off-site links, often of good quality.

Black hat SEOs have cracked their way into these websites ever since people began looking for off-site link authority. Search engines vigilantly lookout for links that appear out of place and remove the ability to pass PageRank from those pages. It makes it easy for Google and Bing to identify potential link spammers too.

If you can get a genuine link from a .edu or .gov, by earning it, do so! Keep an eye out for opportunities on these sites, but do not force your way in with hacking or trickery.

Links in Content

Content links are the best links on the Web. These are links to your pages from quality blog posts, content pages, resource lists, white papers, eBooks, Word documents and other file types the search engines crawl and index.

We love to talk about natural links and you can read plenty of articles on content marketing and influencer marketing or link bait. All content links need not be 100% organic and unsolicited. Getting natural content links is spot-on goodness, but you need to be strategic and tactical too.

For example, before you publish an important article or link bait, email your blogging friends to give them a heads-up and ask them to write something about it. If you do not have blogging friends, reach out to bloggers and websites you admire and build a network from scratch so you can turn to each other.

Just as you help members of your network, and they help you, offline, build and use your online network. Trust me, the top bloggers email each other before they publish content they want others to share. You still need to publish quality content and your network cannot repost or parrot what you write. They need put an original spin on it when they link to you.

Further Reading

• Matt Cutts Interviewed by Eric Enge
• Matt Cutts On Penalties Vs. Algorithm Changes, A Disavow-This-Link Tool & More
• What if links aren’t as important as you think?
• Learn link lingo, and the low-down on link lust
• Read about the Circle of Concern vs. the Circle of Influence in Stephen Covey’s 7 Habits of Highly Effective People

 Distribution curve image from Wikipedia