I’ve had the pleasure of working on SEO in a variety of companies both in house and, frequently, through casual advice to other companies. After eight years in the business, I’m convinced that the success of search for an in-house person hinges on their ability to work within a leader’s approach to search. This is due to the interdepartmental cooperation needed to effectively run search as well as the fluid nature of our business.

What follows are four extreme leadership caricatures, vis-a-vis search, and recommendations on how to most effectively work search into an organization within a specific leadership perspective regarding search leadership styles.

Hands Off Leaders

Image via Shutterstock

The hands-off leader is usually old school and has been successful either prior to the advent of the Interweb or through other online channels such as highly measureable PPC, email or display. Mr. Hands Off probably still has an AOL email address for personal use.

In these organizations, search probably doesn’t exist as a function, and if it does, it’s probably grouped under an entry level PPC or display person. The Hands Off leader has either never decided to push search as a channel or worse (and more likely), has proactively chosen to steer clear of it.

How To Manage Hands Off

This is actually a great opportunity for a search marketer – as the Hands Off leader probably has built a successful business ripe with low hanging search fruit. Your challenge, here, is to get some early easy wins and report metrics up as widely and as high as possible.

Use business metrics such as ROI, cost of sales, cost of acquisition etc., to compare search against other channels. Search will (almost) always outperform.

The biggest challenge in a Hands Off organization is the cross-departmental cooperation that must happen for most search efforts to be effective.

Conquer this by over communicating and always awarding the success of those aforementioned business metrics to other departments. It’s amazing how you can make friends by turning cost centers into profit centers.

Confidently Ignorant

Image via Shutterstock

The Confidently Ignorant leader is a garbled mess of buzzwords and directives. This leader is usually someone who has worked at, but not on an online property and wouldn’t know the difference between a canonical tag and a magic wand.

While she speaks confidently of GoogleJuice, the nuance between linkbait and spam is a mystery to Confidently Ignorant. You’ve never heard her say, “I don’t know,” but she frequently spouts off on dated information she picked up at a networking event that was trendy months or years ago. “We need more Pinterest Juice!”

Usually, nothing substantial ever really happens under Confidently Ignorant.

How To Manage Confidently Ignorant

The good news is that Confidently Ignorant knows that search is important – just not what to do about it. The primary challenge, here, is to focus on those things that do move the needle and minimize the garbage. Use Confidently Ignorant’s bluster to gather resources across departments to push through your search agenda.

Managing in this organization requires a disciplined approach to project management, i.e., identify business objectives before a project is undertaken and then regularly go back and post-mortem every project. This provides you with a structure to demonstrate (privately) that Confidently Ignorant’s  pet project that she crammed through wasn’t a great investment.

In fact, over time, these post-mortems, may serve to simultaneously educate and bore her with the technical details of search. Victory. Unlike every other situation – ascribe search success directly to Confidently Ignorant (instead of the departments who actually did the work) to garner further support for projects you want to undertake.

Aspiring Growth

Image via Shutterstock

The Aspiring Growth leader often runs a start-up, is confident that his product is desperately needed and is certain that the Interweb will deliver a flood of business; he’s just not sure how. Aspiring Growth has picked up a few things here or there, but knows what he doesn’t know. He is admittedly (and appropriately) focused on his area of expertise (usually building out a product that people don’t even know they need yet.)

Additionally, he has limited understanding of the time and competitive factors around search – i.e., it’s going to take a lot more time and resources to get traffic for “mesothelioma lawyer” than “pink fuzzy bunny slippers.” He probably thinks of search in terms of ranking reports.

How To Manage Aspiring Growth

Aspiring Growth needs a lesson in reality. I’ve found that the best way to work with Aspiring Growth leaders is to overwhelm them with education around the technical and tactical components of search, communicating this early on and often. This does two things:  1) grounds them in reality that search isn’t an immediate payoff, and 2) bores them with the details. This provides them with the confidence to let you do your job, while they return to what they do best.

Aspiring Growth is highly susceptible to unscrupulous or just really bad search and Web development agencies, and you may find yourself spending most of your effort unraveling legacy issues that were offshored.

SEO Maven

Image via shutterstock

The SEO Maven leaders are few and far between. She’s probably has a high-end technical degree and sold her first business, that she coded entirely on her own.

The SEO Maven has inculcated search best practices across the organization, does nothing at all to the site without A/B/C/D/E . . . testing, and has created some impressive in-house reporting tools that she replaced Google Analytics with because she’s paranoid about sharing site data. She could probably sell these tools to the agency world, but would never consider it as she’s too focused on using them to build her business.

How To Manage the SEO Maven

This is both a hard and easy place to be as an in-house search marketer. First, there is no way you are going to be the search subject-matter expert, so eat a humility pill every day at breakfast. However, working for the SEO Maven minimizes the political battles and interdepartmental struggles that so frequently occur otherwise when dealing with search.

In SEO Maven organizations, “what have you done for me lately” is the overriding perspective. Test, test, test. Be creative and then test again. These businesses are the ultimate training grounds for search marketers – if someone was getting out of college who wanted to get into the business, this is where I’d recommend they go (especially instead of the agency route.)

Due to the connected nature of our business, highly political corporate cultures tend to flounder in their search efforts while focused, nimble companies often succeed. While we may bemoan this reality, as search marketers, learning to function within the leadership and political structure of our organization is a necessarily skill.

What follows are a series of the most important lessons I’ve (maybe) learned while dealing with MBAs, devs, product managers, designers and my own ego.

Don’t Think Everything Through

We’ve been trained through years of education, higher education (and for some of people,  worse – consulting gigs) to deliver beautifully polished, thoroughly thought-through project plans. We wrap these plans in detailed reports and colored coded timelines and then present them to key stakeholders. Nothing could be less effective.

These reports are inevitably met with questions you haven’t thoroughly thought through (or even considered), staffing limitations, technical impossibilities, competing priorities and political agendas. Avoid this by involving stakeholders as early as possible and having hard, messy conversations early in the process instead of at the end.

Don’t Design Anything

It’s very easy to have opinions on design. Remember what they say about opinions and leave your designers alone to be the professionals that they are.

Don’t Trust Salespeople Who Describe Easy Implementation

I’ve been doing this for 15 years, and I’ve never seen an easy, seamless implementation. Get your technical people talking to their technical people before you sign the contract. Good salespeople will push this for you. Bad salespeople will send you a contract first.

Don’t Assume Noobs Understand Anything

I once watched a new hire struggle for weeks until we had a coffee conversation about search and it became clear he didn’t know the difference between a title tag and an H1 and thought a sitemap was something optometrists used.

Bad hiring process for sure – but you can minimize this problem by pushing new hires through a search orientation, in which you lay out search fundamentals as well as your company’s overall search philosophy.

Don’t Think About Traffic

Ahhh the monthly UU count – that standardized pissing contest yardstick by which all sites are compared. The UU count is a hold over of the ad supported model. It (frequently) means nothing to your company’s business health.

Unless you are pushing Sealy Posturepedic display campaigns or have no ethical qualms in targeting poor college students with Chase Bank credit card ads,  UU’s are probably not the right yardstick for your business.

A retailer, for example should much rather increase converting traffic by 10% than doubling non-converting traffic. This has huge implications for your search strategy.

Don’t Hire People With A Blackhat Background

This is an unfair, broad-brush-stroke generalization, but people who have worked in a “we’re smarter than those massively capitalized search engines” mode have trouble leaving this perspective behind. This is true for both in-house and, even worse, consultant SEOs.

Yes, there are some (in fact many) of the search celebrities who once made tons of money pushing Viagra from Canada who have now been reborn as virginal white hats, but I fear an arrogance that just can’t be left behind.

Don’t Let People Who Did A Project Evaluate It

We have access to more data than we know what to do with. This means that with some good data mining, it’s rare that you can’t shape an analysis in whatever light you want to in order to impress your genius upon a boss.

Avoid this problem by clearly calling out success metrics, various data sources, and the evaluation time period at the onset of the project. Better yet, have all analysis performed by a dedicated, disinterested number cruncher. Speaking of which . . .

Don’t Hire Optimistic Analysts

Optimists make good cheerleaders and visionary CEOs – they make really poor analysts and financial prognosticators. We recently hired a full time analyst who is downright cranky – suspicious of all assumptions, critical of growth multiples, and highly skeptical of any hockey-stick graph that would put a smile on a VC’s face. Best hire ever!

Don’t Follow Ranking Reports

I’m still seeing business leaders obsess over ranking reports and search “tools” that sate the desire to compare rankings for specific terms despite personalization, social, constant changing SERP pages and the fact that this term-based focus can lead you down a very dangerous path.

One cranky day, I wrote a diatribe against ranking reports: Excuse me While I Have a Ranking Report Rant.

Don’t Use Consultants

I’ve made many unfriends (and been uninvited to a few conferences) for pushing businesses away from search agencies.

Ignoring the proliferation of hacks masquerading as search gurus, here’s why search should be an in-house function:

• Given the potential downside, I’d only hire a consultant if I knew exactly what they were doing, and if I knew exactly what they were doing, I’d do it myself.
• SEO touches so many parts of the organization – this is very hard to deal with as an in-house, and it’s almost impossible for an offsite third party.
• SEO changes constantly, so it is vital to have search fundamentals baked into your ongoing marketing, content, and development cycles.
• SEO done well is a constant process of testing and retesting and incorporating the learnings from these tests into the mental memory of the organization. This just can’t be accomplished from the outside.

Good reasons to use consultants:

• If you really don’t know what you are doing and you want to bring someone in to train your staff (or to keep your staff up to speed on the cutting edge issues: “how should we think about the next iteration of Penguin?,” for example).
• If you are doing a highly technical, infrequent change – like a complete change in back end infrastructure.
• Getting a third party audit coupled with in-house training may serve to uncover some things your in-house group hasn’t thought of.
• If you need someone to throw under the blackhat bus – some companies want an agency on record to take the fall when they are inevitably penalized. Remember JCPenney, anyone? SearchDex, the agency they threw under the bus, quickly pulled down their page listing their clients and now, almost two years later, still won’t ID a single client on its website.

Don’t Benchmark Competitors

Trying to emulate competitors will push you to be just as bad as they are. Instead, benchmark best practices (or creative approaches) outside your industry. I’ve written more about this here:  Aspirations of Incompetence:  Benchmarking Competitors.

Don’t Take Credit For Any SEO Successes

Bill Gurly from Benchmark Capital described search traffic to me as “free beer.” As the in-house SEO, you are in a unique position to apportion this keg of free beer across the organization. Nothing engenders more buy-in for the importance of search than public attribution for success.

So, as an in-house SEO, your golden rule is to never ever ever take any credit for anything good that ever happens with regards to search.

Turns out, while search has come a long way, many execs still use terms like “Google Juice” and define success as launching a PPC campaign to “rank number 1 for our competitor’s name”. This issue is even more pronounced in larger, established companies where search makes up a smaller portion of the marketing mix.

:::sigh:::

Jonathon’s primary recommendation centered around “data visualization” – explaining and reporting on search concepts (and progress) through pictures instead of technical jargon and theory.

To the extent that you can translate your SEO efforts into picturebooks for MBAs via powerpoint, you can successfully focus those with limited search understanding on the correct tactics.

Enter Scattergraphs

What we are all really trying to do is develop a clear understanding of “if I do X, then Y is going to happen”.

In mathematical terms, this is called a correlation coefficient – i.e. the extent to which two series of datapoints are interrelated. Correlation coefficients range from +1 (perfect positive correlation) to -1 (perfect negative correlation).

This can get infinitely more complex when you add more than two datapoints, the analysis is a statistical methodology called multiple regression analysis in which you try to determine the extent to which multiple data points impact a variable.

This is the process undertaken by some search consultancies and tool providers who try to use data to backdoor their way into search engine algorithms. Multiple regression analysis is a hairy process, involves words like heteroscedasticity and requires either an advanced degree in statistics or econometrics to do with any degree of accuracy. I stay away.

One note of caution:  correlation does not mean causation. Just because the two datapoints have a similar pattern, doesn’t mean one influences the other. An  obvious example of this is sunrise and eating breakfast . . . while these things often happen in synch, eating your Cheerios at 4 am will not make the sun rise any earlier.

Simple regression, in which we are just looking at fit between two data points is, in fact, pretty easy stuff.  The concept is fairly simple – calculate a straight line that best fits two data points when plotted on a graph. If you want to geek out on the math behind this try the Simple Linear Regression page on this awesome site I just found called Wikipedia.

Here’s a visual explanation of correlation coefficients and simple regression:

(Obviously, this is not my graphic – do you think I’d deliberately highlight a negative correlation between hair and time?)

If you’d actually like to do it instead of remembering the greek symbols behind math formulas . . . use good old Excel. Here’s how:

1.  Select Two Datapoints

While you can calculate correlate between all sorts of things, may I suggest starting with inbound natural search traffic and some variable that theoretically impacts that?

To get multiple datapoints, you’ll need to segment your data – in the case of Urbanspoon, it’s pretty easy – we can look at traffic by city, cuisine type, or entry categories (restaurant pages instead of city pages for example).

Now, normalize that data:  if you are looking at differences by geography, calculate penetration by dividing your entry sessions by population; if you are looking at differences by product category, calculate penetration by dividing by overall search impressions. (Depending on your data sources, this normalization process can be persnickety and tricky.)

2.  Open Excel

Put your two datapoints into two excel columns.

3.  Correlation Coefficient

Calculate the correlation coefficient between the two columns using the CORELL command. This will give you the mathematical correlation coefficient indicating the extent to which those two datapoints are correlated – the closer to 1, the more tight positive correlation, the closer to -1, the more tight negative correlation. Correlation coefficients close to zero indicate no correlation.

4.  Turn this Number Into a Picture

Use excel to create a scattergraph of these two columns like the ones above. I like to put the natural search penetration on the vertical axis and the tactical variable on the horizontal axis. Assuming there is a correlation . . .

5.  Impact the Variable

Engage in whatever tactic are analyzing by selecting a few of the datapoints that are underperforming (i.e. for positive correlation, these datapoints will exist in the bottom left hand quadrant of your scattergraph.) This tactic can be linkbuilding or social mentions for example. Your goal is to move the datapoint along the horizontal axis and see if it also moves up the vertical (penetration) axis.

6.  Wait

How long you wait depends on what tactic you are using and how quickly (theoretically) you think it’s going to take for the tactic to have an impact.

7.  Redraw the Scattergraph

Now, after you have a new set of data, redraw your scattergraph. Highlight those variables in a before and after comparison of the scattergraphs and demonstrate to your MBAs the extent to which movement along the horizontal axis is reflected in movement up the vertical axis. Highlight this movement with arrows or different colors for your test datapoints.  Y

ou can even redraw both data grabs using different colors on the same graph, or show a simple before and after.

8. Declare Success or Failure of Tactic

The result being to roll out your effort more broadly or abandon the tactic altogether.

This gives you a real way of calculating the impact of your tactics. If you have cost metrics (and you should), you can transcend discussion of GoogleJuice (yummy, I like mine on ice) and make ROI driven investments in search.

Despite missing the entire point of the article (that you can use simple free tools to calculate confidence intervals for A/B split tests), Eric did raise an interesting challenge:  “are you going to let them go online and use this tool at the interview?”

The answer? Of course I am. Why the hell wouldn’t I?

I wouldn’t expect anyone to do their job without access to the Web, so why on earth would I conduct an interview devoid of a computer and Web connection? That would be liking hiring a jockey without seeing him ride or a heart surgeon based on her MCAT scores.

If you really want to assess how someone works, attach a laptop to a projector during the interview and give them free reign to use the Web as they see fit. You’ll see proficiency with Excel, intimacy with third party tools and quickly build a map of how their mind functions. Nothing serves better in separating theory from practice than watching how someone works.

Over the past 7 years working in search, I’ve refined my interviewing process down to a two page cheat sheet. The result – I’ve hired some of the best co-workers imaginable. Smarter than me. More technical. More detail oriented. More experienced. More creative.

Depending on which side of the desk you are on, what follows is either a “how to nail a search interview” or “how to hire awesome search talent.”  Note that this is admittedly through the heavily biased lens of a business guy within an aggressive mid sized business – one size won’t fit all.

My first question is always a math problem. This sets an immediate precedent that the job is highly analytical and weeds out the marketers who would rather talk about what kind of tree their brand would be if it were a tree.

It also lets me assess if this is going to be a quick, polite 20-minute conversation or an in-depth hour long dialogue. I actually watch the person’s physical reaction to the first question, if he or she is uncomfortable with analytics its going to show. Any answer that hints “I’m not really good at math” = 20 minutes and a handshake.

Below are a set of questions I draw from:

Question 1

“You are A/B testing two different ad creatives . . . here are their impressions and clicks. Imagine on your first day, I tell you to optimize the campaign of 50 keywords, what do you do?”

Better yet, put data in front of them from a small Adwords account and just ask a simple, “What do you do?”

While this isn’t natural search specific, its an easily understood simplistic pre-Algebra problem. The (insufficient) math-only answer is to calculate CTR and declare a winner (per EricM above).

Years of schooling have trained us to identify a math problem, calculate the answer and move on to the next problem. This is the wrong approach and yields the wrong answer.

What I’m really looking for is some holistic, out of the box thinking that is driven by a business orientation and technical experience.

The right answers (yes plural answers) include:

• Consideration of sample size and confidence intervals.
• Looking downstream for differences in conversions in the creative.
• How can I further segment the 50 keywords to get more granular groups.
• For anyone with any PPC in their background, I’d expect some discussion around Quality Scores and landing page optimization testing.

Question 2

“In the above example, calculate the ROI of Ad A.”

This is a great, albeit obvious test for business orientation. The engineer who can get this question is surprisingly rare, but amazingly valuable. A “marketing person” who can’t come up with the basic formula isn’t going anywhere.

Also note that the data I’ve provide the candidate so far doesn’t include anything around revenue or costs. I’m looking for the candidate who will quickly ID those missing elements, then put them together.

Question 3

“How would I use an analytics package to track reservations made on Urbanspoon?”

I’m looking for an understanding of how conversion tracking works – ideally within a large analytics package, like GA or Omniture.

Question 4

“According to Comscore, Company XXXX seems to been outpacing our growth in search over the past 12 months, why do you think that’s the case?”

I’m looking for answers that utilize some of the paid or unpaid tools on the market. I’d expect someone to recommend doing a comparative link evaluation using tools from Majestic, SeoMOZ, Raven or even something free like Blekko.

This is a great time to let the candidate actually log on to a tool and walk you through how they use it. Having a Majestic account is very different from being able to use one; in the same way that having a gym membership has not slowed the progression of my increasingly soft midsection.

Question 5

“Tell me about Caffeine or May Day.”

I’m not looking for my favorite beverage or descriptions of young girls dancing with ribbons. These are two named algo updates from 2009/2010 and familiarity with them indicates some duration of hands-on search. If you are looking for someone with serious longevity, ask about the Google Dance.

Question 6

“Here is an excel file with data for all of the 600,000 restaurants in the United States from Urbanspoon. Turn this file into a list of cities with more than 1,000 restaurants.”

The answer involves pivot tables and should take about 1-3 minutes.

Question 7

“Now, imagine we purchase a list of restaurant emails and want to append our huge database with those emails … how would we go about doing this?”

The answer involves a VLookup, which can be persnickety and tedious – fair to assume if they know the answer, they can work through the actual implementation.

Question 8

“We’re testing television advertising to evaluate our ability to extend our brand from a directory to a reservations destination. How would you evaluate the ROI of this?”

This question requires a fairly complex answer and I’m looking for nuances that include:

• What is your business objective?
• How is the test set up . . . what is the control I’m comparing this to?
• Some concept around lift and then extended lift (this is more of a branding answer, but ties into the notion of multiple touch points instead of focusing entirely on last click attribution – which is a very common mindset.)
• Search bonus for looking for lift uptick in branded search as well as direct traffic.

Question 9

“How can we rank better for ‘Romantic Italian Restaurants in Redmond’ – here’s the page, but it ranks on the second page for that term.”  (Assume said page has standard on page optimization in place.)

This is a red herring question - what I really want is for someone to tell me something along the lines of:

• “ranking reports are a waste of time and here’s why”
• “you should look at the inbound search traffic instead of the ranking report for a specific term”
• “this is a long tail term, if you spend time chasing every long tail term, you’ll miss the forest for the trees.”

Question 10

At some point, I’m going to dig really deep on something where I have a lot of experience (say local or news search) where I know the candidate is going to hit the ceiling of her knowledge.

In this case, the right answer is, “I don’t know.”  There’s a lot of changes in search, making it impossible to know everything. Being able to tell your interviewer (or boss) “I don’t know” is very important. There are already enough people trying to BS their way through the search industry and I don’t want them working for me.

Conversely, (and just as importantly) I want the candidate to enlighten me. A great candidate should be able to teach me something.

If all else fails to draw this out, try:  “Tell me something I don’t know about search.”  The close cousin of the tell me something question is forward looking: ”what is the next big thing we should be thinking about?” or “My CEO is all atwitter about Pintrest, how should this play into our search strategy?”  

Question 11

“Tell me what you know about me.”

This isn’t a narcissistic probing into my favorite topic; but I’m really asking to see if a) the candidate has done any preparation and b) if the candidate is social. The integration of social into search makes this question increasingly relevant, especially for smaller entrepreneurial companies that may not have a “social” department or dedicated function.

Smart candidates avoid this question by dropping hints about the interviewer’s background during the interview: “I saw you played rugby”, “lived in Ireland”, “have kids” to let the interviewer know they’ve done some research. This can also take the form of, “In the New York times article last week about Urbanspoon…”

Question 12

Finally, the best question in any interview is a series of “whys”. I interrupt repeatedly with why.

Why? Because it challenges statements. Why? Because some candidates try to gloss over hard questions with assumptions. Why? Frequently, I find explaining assumptions leads to discussions on fascinating tangents. Why? Because their experience is very different to mine and I want to know what they’ll bring to the table.

Onerous interview process? Uncomfortable? Maybe. But, I’d much rather have a tough hour long interview than a tough month or two or six with a poor fit. And let’s be honest, anyone worth his or her salt is going to have to answer much harder, on the spot questions across the organization every week.

For more on hiring awesome SEOs, check out Luc Levesque’s post:  How to Sniff out Rockstar SEO Talent.

“Let’s say this is your first day at Urbanspoon and I show you the following data. We’ve just launched an A/B test of that I’d like you to evaluate. [The example can be almost anything you want to test different results for – from almost any search element, PPC campaigns, email subject lines etc. In this case, I’m using a PPC example.] Imagine you are running two different ads on a campaign with 50 kewords. We’ve been running Ad A for a while and have 17,235 impressions and 272 clicks. I started running Ad B last week and that has received 41 clicks on 2,253 impressions. What would you do?”

I’m looking for an answer that goes beyond demonstration of pre-algebra skills and rudimentary familiarity with a calculator.

Obvious answers include splitting up 50 keywords into different groups, looking down stream to see differences in conversion rates, and technical answers around quality score. But what I’m really looking for is a theoretical understanding of statistics and the interplay between sample sizes, variability and confidence intervals.

Answers to the above theoretical question usually fall into one of three buckets:

1. I’d run as more Ad B’s so our impressions are equal and then compare the click through rates.  #FAIL
2. I’d run the Ads longer, you need at least 3 weeks of data to make a decision.  #FAIL
3. Ad B is better b/c the click through rate is higher.  #FAIL “and thanks for taking the time to talk with me, our HR department will be in touch . . .”

Turns out, you don’t need to have an equal number of impressions or a set amount of time to run this analysis. It’s actually a fairly simple concept that can ultimately then be mathematically defined:

The greater the difference between your A and B samples (drawn randomly from the same pool) the smaller the size your test needs to be in order to confidently assert that one performs better than another. Example – if we wanted to test if men were taller than women and we measured 100 men and 100 women  and the men averaged 7 feet tall and the women averaged 4 feet tall, you’d be fairly confident saying that men are taller than women.

Conversely, if the difference was 3 inches instead of 3 feet, you’d probably want to measure more men and women before confidently asserting men are taller than women.

In fact, it’s possible that your sample was misleading – as a population women are really taller than men, but your sample didn’t bear that out. This level of confidence can be mathematically expressed as a percentage – I’m 95% certain that A is better than B. (Meaning there is a 5% chance, or 1 out of every 20 times where you’ll unwittingly pick the underperformer.) The greater the level of confidence you want, the larger the sample size you need.

All of this can be calculated with innumerable free online tools. Larger sophisticated systems like Adwords and big ESPs build this statistical testing in to their testing methodology – but it’s easy for do-it-yourselfers too.

I like a tool called AB Tester, which allows you to measure up to three alternatives compared to a benchmark:

In the results above, I’ve done the analysis for our question . . . The “Confidence” column tells me there’s a 79.19% chance that B is better than our control A.

Watch how this Confidence grows when we add a zero to each column – keeping the CTR the same but increasing the sample size:

By increasing the size of the test tenfold, now there’s only 0.5% that A is really better than B.

Let’s go from theoretical to real. Here are results from an email test we did for our Hawaiian getaway promotion to Ludobites 9. (It’s over now, sorry.)

The first data column is sends, then delivered, then opens, then clicks. Assume we want to test three different content types to three different cities (now admittedly this is not a random sample – maybe people in San Francisco respond differently to content . . . )

Take the data from the 3/6 send and plug it in to A/B Tester. Note I’m comparing the CTR from Opened emails to isolate content as an impact to click through rate. Also note that while the sample sizes are similar, they don’t have to be the same.

My best performer here is the San Francisco content at a 5.5% CTR. I use that as a control and plug the other two into AB Tester:

This tells me there’s a 3.3% likelihood that the LA content might really outperform the winner (San Francisco). Additionally, there’s a 23.5% chance Seattle content is better than our “winner”. More testing necessary . . . .

Companies are increasingly integrating user generated content, reviews and social media into their overall marketing initiatives. In principle, this seems like a good idea. In practice, it can falter.

I’m going to review the search implications to proactively (or not) incorporating user generated content into marketing efforts using three unlikely examples:  KY Jelly, Susan Komen and Jesus.

Intense Effect – KY From Johnson & Johnson

In recent years, KY brand, owned by Johnson and Johnson, has engaged in a successful brand extension into additional adult products. Their goal is clearly to bring those products generally only available in stores with blacked out windows to the masses. Product packaging is understated. Later night TV spots feature everyday couples expanding their comfort zones. Consumers can pick many products up at traditional retail outlets like drug stores.

Recently, KY introduced KY Intense (a warming lubricant) that was packaged in a tiny potion-like bottle (instead of the traditional clinical white tube that looks like it could hold anything from toothpaste to plumbing caulk and is named “Jelly” which sounds like, well, ewww…) So far, so good.

To support the product, KY launched a series of ads including a clever euphemism spot called nutmeg and even a same sex couples spot. The ads featured everyday couples in their homes, describing how the product vastly improved the woman’s climax.

Click here to view the embedded video.

Now of course, this is a very big promise – like guaranteeing unicorn sightings or Herman Cain’s reentry to the GOP ticket. The ads end with the following tagline:  ”scientifically proven to make that big moment feel even bigger”, essentially saying, “if you can’t be satisfied with our product, even PhD’s can’t help you.”

But it gets worse.

Recently, KY launched a social media driven site to support KY Intense with a primary objective of soliciting and sharing user reviews. New ad spots end with a call to action to go to this site, which features a screen sized pop-up of recent product reviews and a call to action to submit your own.

Simple buttons make Facebook sharing easy – although, do I really want to share this w/my mom?

If you are spending millions of dollars to drive people to a site that is full of users reviews, you’d better make sure that you have a great product. Unfortunately, this is where KY Intense falls down.

Five of the five are negative. So you don’t have to squint, I’ll copy some of these down for you:

This is just overpriced massage oil

I was really dissapointed in it. . . i hate he wasted his money; he really thought the stuff worked

Didn’t work very well

Didn’t change much, more of a placebo than anything else

I haven’t tried it yet because I saw how expensive it was. KY also has a warming liquid which is alot cheaper?? I want to know the difference between the 2 products?

As you can see, there are very few truly satisfied customers – in every sense of the word. To their credit, KY offers to expose “the good and the bad and everything in between” about their product, but this is hard with such lofty expectations. With user generated content, when you promise unicorns, you’d better deliver a damn unicorn.

Also worthy of note – the majority of reviews have been positive, but because they are displayed chronologically, the current view looks extremely negative. Even a navigational search for “intense effect” returns pretty negative results directly on the SERP:

And those little sitelinks below? Really want to learn what mindy c. has to say and you’ll land on a completely unbranded, text only page. (The “bad user experience” double entendre jokes keep coming . . . )

Komen Clams Up (In Search)

As a counterpoint to this, consider the recent Susan G. Komen public relations disaster which was driven extraordinarily heavily by social media channels. I was curious to see search results during the PR maelstrom.

The following screenshot was taken on February 2, 2012, at the height of the controversy and with the noted exception of 3 news results, everything on the SERP page was either generic or controlled directly by Komen. Imagine if those Komen pages included metadescriptions of unfiltered user generated content.

Komen’s very very strong search presence combined with a lack of any user generated content in the search results enabled them to manage (at least is search) the PR disaster. (Interestingly, a month after the bruh-ha-ha subsided, I’m showing the last four results on Google’s SERP covering the news.)

(Note: that search results might look differently to you depending on localization.)

Jesus Is . . .

You’ve undoubtedly seen the “Jesus is _______”  bumper stickers – white text on a black background designed to make you consider your own approach to religion while you sit in a traffic jam. It’s pretty clever, actually – you’re a captive audience during a time when you can just let your mind wander.

Well, the Jesus is campaign has taken that introspection to the user generated level: I’ve started seeing Jesus is ads popping up on Facebook.

(I know the targeting for this is pretty broad – given a few patriotic like’s and my past working in the legal industry, Facebook thinks I’m either ex-military or a soon-to-be-deported, drunk driving, victim of a pharmaceutical negligence whose wife is leaving him.)

Search results for “jesus is” proactively solicit user generated content submissions:

Now, of course, the very purpose of the “Jesus Is” campaign is to spur introspection. There is probably not a single topic that could have a wider variety of highly impassioned responses. The homepage for Jesus Is is matrix of constantly updating answers to the Jesus Is question:

Jesus is a phony!  My Hero?  Homophogic Bigotry?

By publishing thoughts across the spectrum, the Jesus Is campaign pushes viewers to not only think about their own perspective, but to understand a wider perspective – and this, I suspect is the ultimate goal of the campaign.

The City Church (the people behind the campaign) won’t just publish any fill-in-the-blank. Note on the submission form below:  ”submissions will be reviewed before they are published to ensure they comply with the Terms & Conditions.”

My industry joke below will probably never see the light of day, nor will anything completely offensive.

Through this moderated hybrid approach, City Church effectively uses user generated content to push their agenda.

Every company must determine how proactive they are in incorporating user generated content into their search strategy and user experience, but I suspect we will see more and more of this in the near future.

The end result confirmed what I had long anecdotally experienced – the difference in result relevancy between the engines was really not that much. In fact, Bing bested Google slightly.

Admittedly, my little test, with n=20, had the academic rigor of a Sarah Palin geography lesson. I was rightly skewered by some readers, including one who called the article a “shitpost”.  But the concept of quality parity between the search giants was so unexpected that some other media outlets picked up the post.

Among all the comments, this one, by Cathy Reisenwitz answered the big question, “if there is such little disparity in quality, why  is there such huge disparity in market share?”

“People aren’t switching to Bing because Bing needs to be much better than Google to make it worth the switch.”

Time to revisit the question – is there really a big quality difference between Google and Bing?  Over the past 12 months, many things in search have changed.

We’ve seen an unprecedented reversal in transparency from the engines, the explosion of social as a factor, modest Bing marketshare gains, stronger anti-competitive allegations and a (an?) hiybbprqag search dragnet.

Google vs. Bing:  Round II

The objective was to test and the search engines’ ability to deliver quality results, admitting that quality is a very subjective term, but includes things like timeliness, 1 click access to info, volume of content and lack of spam.

The Rules

I expanded the data set from 20 to 100 – collecting results over the past 4 months or so on real search terms from yours truly – (thus the inclusion of “clean crayon off a lcd t.v. screen”). Again, I included terms that were obscure, or could be difficult for engines to handle.

For example, “attorney Tom Brady” meant I was looking for an attorney named Tom Brady, but a logic-driven computer might confuse that for interest in the hunky quarterback’s legal troubles.

I also wanted to capture any personalization so stayed logged in to all accounts (which should have given Google a slight edge.) Paid results were not included in the evaluation set.

Scoring

• A number one result earned 5 points, top 3 earned 3 points and a page one showing earned 1.
• Failure to show up on page one cost an engine 5 points.
• 2 bonus points were awarded if the answer to the question was delivered on the SERP page.

An additional 2 bonus points were awarded if at least one of the top 3 results was from an authoritative site, as determined solely by yours truly . . . Wikipedia in, eHow out.

Sampling The Search Data

Interestingly, there were many queries that didn’t return any results. This was certainly due to user error (but is that really the user’s problem?). But in many cases, the content just simply (and surprisingly) didn’t exist on the Web.

For example, Salomon, sells a full line of ski boots in a highly competitive industry. The only explanation I could find for the stiffness index difference between the Salomon Impact 120 CS and the Salomon Impact 110 CS was by subtracting 110 from 120. Hmmmm – not much info to go on when dropping $500 on a pair of boots.

Likwise, the answer to “who was Kim Jong Un’s mother?” was surprisingly difficult to find.

I remain amazed that neither Google nor Bing has really figured out the people/address side of search – my default when looking up someone’s address is still a direct load of 411.com; despite the fact that site invariably tries to upsell me on some affiliate cyberstalking product.

This is particularly painful when updating a Christmas cards list. Perhaps this is the engines’ deliberate decision to mollify those concerned about privacy issues, but it is very frustrating.

I ran some searches that had an unwritten time element to them, like “wii new release rumors”. These searches are particularly difficult for engines to understand the users’ intent. In that case, I was wondering if I should delay a purchase, instead of researching past rumors about wii launches.

For the query, “distribution of type of search engine queries”, Google sent me to Wikipedia, but unfortunately the data on there cited studies from 2001. Likewise, “what percentage of people online use Twitter” – Google sent me to current data while the only results on page 1 from Bing went back to studies from 2010.

The most interesting result was for the query “perpetrators behind rick santorum googlebomb”. I’m not sure if it was the inclusion of “googlebomb” or “santorum”, but Bing initially gave me a no results page. (Although when I checked it again while writing this, they served up Danny’s post on Santorum.)

Another one that begs for a conspiracy theorist was Bing’s results for “google analytics import match type data from adwords”.

Page 1 was littered with anti-Google sentiment, including headers like: “Comparative Analysis:  Omniutre SiteCatalyst, Google Analytcs” and “STOP Google Analytics from Stealing your valuable Adwords Keyword data“, neither of which have anything to do with match types. Did we catch Bing favoring certain results? Or is this a natural occurence?

I could smell the impact of Panda in the results . . . . Bing seemed to heavily favor some weak UGC sites like eHow and a chacha, (which looks like the ugly lovechild of Entertainment Weekly and Quora.)

My favorite garbage content came on eHow – for “how do you change the water filter on a frigidaire professional series”.  “Step 1:  Open the freezer or refrigerator door . . . ”

Having said that, Bing’s overall score was buoyed by a slick incorporation of a travel widget for all airline related queries.

The Results

Same as last year – a statistical dead heat; meaning overwhelming parity between the engines.

Google:  296

Bing:  274

While the marketshare and tech sentiment suggests the score should have looked like the college football West Virginia beatdown of Clemson, it looked much more like Romney squeaking out a win over Rick, errrrrrr Santorum in Iowa. And like Romney, this isn’t a win for Google.  I suspect we will continue to see Bing make very slow inroads during 2012.

One final note – you can see the raw data by downloading the Excel spreadsheet from my test here.

Remember, I ran the searches on my computer at work over a period of four months and you most likely won’t get the same exact results if you recreate some of these searches on your own. In fact, as I reran some queries while writing this story, I got completely different results.

The North Pole

Dear Search Community:

Thanks so much for your many letters to the north pole. Your nicely written texts and emails included some familiar requests from last year and some entirely new ones. Lisa and Ian, I’m very sorry, I can’t drop that referring keyword data  from signed in users down the chimney. And Eric, I’m really sorry that the European Debt Crisis, which was my answer to your request from last year “Give European regulators something else to think about” didn’t really pan out.

So I’m taking time off from wrapping all those iPhone 4s and Kindle Fires to lodge some requests with the search community from Kris Kringle. I’ve been good to you over the years and it is time you helped Santa fix some of these search problems.

First off, I did a little online reputation management monitoring:

“Including results for Christopher”!  Only Mrs. Klaus calls me Christopher, and only when I’ve been naughty (don’t ask how naughty.)

And see that second result . . . for a site called MySpace? It brings good little girls and boys to a promotional page for the album “The Strip Joint”, but a certain imposter named Chris Kringle. The only thing we have in common is a strange fondness for matching velvet suits. Based on what he sings about, he’ll be getting nothing but coal for him and his shortys.

Next, I checked out my local results, and clearly I have a long way to go. Kiddos from all over the world are searching for my address to write letters and this is the best we can do?

And what’s the deal with this site called Wikipedia?  It seems to have the corner on the holiday market.  Here’s “rudolph”:

Even my nickname (get it – “Nick” name?) delivers Wikipedia:

And “reindeer”.  (Please note that I’m very unhappy about the “Reindeer hunting in Greenland” link on this site. Seems the Greenland Board of Tourism has been trolling around in Wikipedia. How can we get rid of that?)

Now for my trademarked term, “Merry Christmas”, we get a 1994 Mariah Carey album on Wikipedia. Are you out of your mind?! Even Mrs. Klaus doesn’t listen to that stuff anymore.

That reminds me, some kid named Bieber sent me a letter back in October asking for that album. Apparently that’s all he wants for Christmas:

I’m extremely annoyed with the results for  ”Ho Ho Ho”. After the two wikipedia entries, I stumbled across something called Urban Dictionary. I’ll assure you this is NOT what I mean when I chuckle merrily:

Fortunately “gingerbread” delivers something completely different (that frankly I don’t understand at all).

I guess I should be happy that with the over commercialization of the holiday season, none of these terms have been appropriated by those black Friday crazy retailers. Seems like a lost opportunity eh? But what would I know…

It makes me wonder though, why there’s not a single ad for the term “Christmas” on Google?  (and yes . . . that’s Wikipedia again.)

Are the engineers in Mountain View holding the holiday sacred? Not the same over at Bing, which helpfully reminds people of the exact date Christmas too. (And yes – there’s Wikipedia yet again.)

On the bright side, I’m glad that my summer side business seems to be doing very well on search.

Be well search community. Perhaps you can help me out this year. In the meantime, remember, I’m not the only one watching the search results to see who’s been naughty or nice.

Happy Holidays!

Love,

Santa

 SEO:  “So, I’m concerned that we’re not proactively link-building and that may have a negative impact on our high quality in bound traffic.”

Executive:  “But when I type-in the company’s name, we are number one in Google.”

SEO:  “Yes, well you are getting personalized results AND there’s a lot more to . . .”

Executive:  “Don’t worry, we have a lot of PageRank juice all over our website.” 

SEO:  “But the amount of converting traffic has declined over the past . . .” 

Executive:  “Besides spiders really like our platform.  Robots do too.” 

SEO:  “OK, I’ll go clean up all that PageRank juice.  Where’s the mop and bucket?”  

The purpose of this post is to provide SEO’s with a series of metrics with which to evaluate your link profile as well as a series of visuals with which to scare MBAs into action. (Or at least approving a budget.)

While I’ve written against benchmarking within your own industry; for internal reporting to business executives, comparing your company to competitors can be extremely effective for galvanizing support and loosening pursestrings.

To make your job easier, I’m drawing data entirely from free tools – primarily Blekko, Majestic and SEOmoz. The former offers SEO data after every search; the latter two offer products: Site Explorer and Open Site Explorer, not to be confused with Yahoo’s . . . . errrr . . . Site Explorer.

For demonstration purposes, I’ll compare Urbanspoon to OpenTable, now that both are in the online reservations space. Accessing these tools is pretty easy and there are more than a few browser plug ins that aggregate some reporting. I use SEO for Chrome Extension for a very quick, cursory look.

Of course, the obvious starting point is a simple overall metric.

Google PageRank is an obvious (and simplistic, poor, inadequate, misleading) choice, that many fallaciously believe aggregates all of that delicious Google Juice into a simple number.

Both SEOmoz  and Majestic offer a variety of metrics on a 100 point scale – Domain mozrank (SEOmoz) and Domain Authority (Majestic) are their respective attempts to calibrate Page Rank.

The problem with overall metrics, of course, is that they tell a very, very small part of the picture.

Instead, let me suggest a variety of supplemental metrics that are both actionable and highly scary (read: competitive) to MBA types.

Competitive Domain Diversity Ratios

I prefer to use domain diversity to evaluate the quality of linkbuilding campaigns. It’s not a perfect metric; however, it more accurately reflects genuine linkbuilding initiatives than sheer link volume. Looking at the sheer number of links entirely misses the point.

To wit – a single site with a footer link could generate thousands, even millions of links, all of which are completely useless.

Get domain diversity from Majestic under the Linking Domains tab – be sure to update the pulldown for “pages on this root” (below). In Blekko, just do a search for your site, click, seo underneath the result and then hit “inbound links” under Domain SEO.

On SEOmoz’s OpenSite Explorer, make sure you look at the entire domain by using the drop down “pages on this root” and then running the report.

Note:  you only get three freebies daily with SEOmoz; at some point, Majestic prompted me to create a free account too, which gave me enough reports to write this article in one day.

Compare these three reporting tools and graph:

Cumulative Domain Diversity from Majestic

Graphically depict differences in domain diversity over time with Majestic’s historic reporting.

This requires a free account with Majestic to get access to their cumulative domain report and graphically compare your site with up to 4 competitors.

Non-Homepage Links

The true art of an SEO’s linkbuilding lies in the ability to drive links to interesting content, not your PR team’s ability to drive stories that link to the homepage (although these links are extremely valuable.)

Look at your homepage to non-homepage link ratio. (You are shooting for the lowest percentage of links going to your homepage here.) I can do this easily with Blekko, Majestic or SEOmoz data.  Here’s the reporting from Majestic:

Just run reports for both the Root Domain and the Page and calculate a ratio of homepage:overall on both links and domains. Below is the data from Majestic with 5 different domains:

Again, if you can look at this from a link perspective, it tells a very different story than the domain perspective. My strong bias is that domain comparison is much more reflective of overall organic link strength of a site.

The next three graphs are based on the assumption that overall link strength is reflected in your long tail results.

Ideally, you should see growth in both the number of landing pages and the number of different keywords brining traffic to your site. You can get both of these from Google Analytics (and obviously can’t get them for your competitors).

Number Of Landing Pages

Number Of Keywords

Sitemap:Crawl:Index:Entry Ratio

Finally, the grandaddy of all data wrapped into a single visual. This reviews the effectiveness of all of your pages at driving traffic. This is based on the assumption that the better your link profile, the more pages you’ll get crawled, the more pages will be indexed and the more pages will get traffic.

You can make pretty scary graphs that would make any MBA Strategy 101 class proud by turning these into stacked area graphs in either absolute (to show progress over time) or relative (to show the percentage of your pages that don’t have a chance of getting any traffic).

Business leaders, frequently molded from concentrated Type-A material, can’t help but obsess over performance relative to competitors.

They celebrate gains in market share, the thrill of stealing a client and the schadenfreude of a competitor’s stock decline. When introduced to SEO, they obsesses over comparative ranking reports.

And yet, none of this really makes any sense from an SEO perspective.

Following a competitor’s SEO lead, especially around specific tactics, can lead you to striving to be just as bad as they are.

Most Industry Leaders Aren’t Industry Leaders Because Of SEO

Generally, the market leader of a given industry (and I speak in very broad brush strokes here) was established well before the SEO game arrived in the corporate board room agenda. “Quarterly Board Room Meeting Minutes: We need to put some more SEO on our website. Motion passed.”

These big, clunky industry leaders’ market dominance comes from decades of business success — so benchmarking (and copying tactics) for SEO doesn’t make sense.

I’ve had the privilege (and horror) of being exposed to SEO discussions of some very large organizations. It is shocking to know just how confused their thinking is and how deep their problems run.

Competitors May Not Use Best Practices

Following the industry leader’s online tactics assumes they are implementing best practices. The reality is, they may experience success for a variety of reasons that you can’t replicate. They are a well-known offline brand; they have a very old, established domain; they have tons of legacy links; etc.

These factors may drive their success regardless of how poor their SEO tactics are or how antiquated their technology platform is. Copying these worst-of-breed tactics without their non-SEO strengths is a very dangerous practice.

Even Market Leaders Play Dirty

The ugly cousin of “not using best practices” is deliberately (or unknowingly) employing black hat tactics. It’s tempting to assume that market leaders employ only white hat tactics. Yet every six months, like clockwork, we hear about yet another well-known, established brand blowing it all.

J.C. Penney anyone? BMW? FindLaw? Of course, big brands can quickly throw their SEO consultancy under the press bus, but why do we think it’s a good idea to copy their tactics? Benchmarking competitors may get you banned. And if you are a small site, this ban may be forever.

Competitors Have A Very Different Link Profile

The aforementioned Type-A business leaders, with enough SEO understanding to be dangerous but not useful, often operate from a simplistic perspective.

How many of you have heard a variation of this: “We’re both in the same industry; they have a PageRank 7 site; they are the market leader; all we need to do is to get a PageRank 8.” 

Presumably, if you are reading this, you already know this is a massive oversimplification, bordering on stupidity. Obviously, the nuances around the link profile for two sites can be very different — number of links, plus ones, domain diversity, deep links, anchor text, links from Digg, tweets, links from Chelsea Clinton’s fan page.

Yet, how many engagements or in-house conversations involve discussion around the PageRank of your competitors?

Ranking Reports Are Useless Competitive Benchmarks

I’ve written a diatribe against ranking reports here, but suffice to say, benchmarking competitors against specific terms is extremely short-sighted.

In summary: Ranking reports don’t take into account search personalization, variations of search queries, the long tail and search volume, and they encourage short-sighted (read: black hat) tactics.

The Solution: Cross Industry Benchmarking

I’m not suggesting that you should never look at other companies’ approaches to SEO. In fact, staying up to speed with the constantly changing landscape requires a healthy dose of watching what others companies do.

The important nuance here is that those benchmarked companies aren’t necessarily the market leaders of your industry.

Consider the following:

When the market research team for gun and ammo manufacturer Remington came back with a study that customers wanted shinier shells, the product development group struggled unsuccessfully to cost-effectively build shiny shell casings.

“Sorry marketing, it can’t be done.”

They failed, that is, until they took a tour of a Maybelline factory, literally across the street, that had been churning out shiny lipstick packaging for years.

When FedEx wanted to improve its on-ground turn-around times, it benchmarked its tactics against the best practices employed by Indy Racing pit crews. Had it looked at the airline industry market leader instead, it would have aspired to be just as bad as its own industry.

The value of cross industry benchmarking holds true online. Want to benchmark best practices for social media outreach? Watch Zappos. Want a benchmark for an awesome approach to user-generated content? Spy on Seeking Alpha. Need a kick in the link building arse? Do a thorough analysis of TripAdvisor.

Hundreds of companies out there are applying cutting-edge SEO best practices. It’s highly unlikely that the leader in your industry is one of them.