I was shocked by the widespread, deeply negative reaction to the introductory paragraph of my column last month, in which I described beginning every interview with a math problem. The response is probably best typified by EricM:
Despite missing the entire point of the article (that you can use simple free tools to calculate confidence intervals for A/B split tests), Eric did raise an interesting challenge: “are you going to let them go online and use this tool at the interview?”
The answer? Of course I am. Why the hell wouldn’t I?
I wouldn’t expect anyone to do their job without access to the Web, so why on earth would I conduct an interview devoid of a computer and Web connection? That would be liking hiring a jockey without seeing him ride or a heart surgeon based on her MCAT scores.
If you really want to assess how someone works, attach a laptop to a projector during the interview and give them free reign to use the Web as they see fit. You’ll see proficiency with Excel, intimacy with third party tools and quickly build a map of how their mind functions. Nothing serves better in separating theory from practice than watching how someone works.
Over the past 7 years working in search, I’ve refined my interviewing process down to a two page cheat sheet. The result – I’ve hired some of the best co-workers imaginable. Smarter than me. More technical. More detail oriented. More experienced. More creative.
Depending on which side of the desk you are on, what follows is either a “how to nail a search interview” or “how to hire awesome search talent.” Note that this is admittedly through the heavily biased lens of a business guy within an aggressive mid sized business – one size won’t fit all.
My first question is always a math problem. This sets an immediate precedent that the job is highly analytical and weeds out the marketers who would rather talk about what kind of tree their brand would be if it were a tree.
It also lets me assess if this is going to be a quick, polite 20-minute conversation or an in-depth hour long dialogue. I actually watch the person’s physical reaction to the first question, if he or she is uncomfortable with analytics its going to show. Any answer that hints “I’m not really good at math” = 20 minutes and a handshake.
Below are a set of questions I draw from:
“You are A/B testing two different ad creatives . . . here are their impressions and clicks. Imagine on your first day, I tell you to optimize the campaign of 50 keywords, what do you do?”
Better yet, put data in front of them from a small Adwords account and just ask a simple, “What do you do?”
While this isn’t natural search specific, its an easily understood simplistic pre-Algebra problem. The (insufficient) math-only answer is to calculate CTR and declare a winner (per EricM above).
Years of schooling have trained us to identify a math problem, calculate the answer and move on to the next problem. This is the wrong approach and yields the wrong answer.
What I’m really looking for is some holistic, out of the box thinking that is driven by a business orientation and technical experience.
The right answers (yes plural answers) include:
- Consideration of sample size and confidence intervals.
- Looking downstream for differences in conversions in the creative.
- How can I further segment the 50 keywords to get more granular groups.
- For anyone with any PPC in their background, I’d expect some discussion around Quality Scores and landing page optimization testing.
“In the above example, calculate the ROI of Ad A.”
This is a great, albeit obvious test for business orientation. The engineer who can get this question is surprisingly rare, but amazingly valuable. A “marketing person” who can’t come up with the basic formula isn’t going anywhere.
Also note that the data I’ve provide the candidate so far doesn’t include anything around revenue or costs. I’m looking for the candidate who will quickly ID those missing elements, then put them together.
“How would I use an analytics package to track reservations made on Urbanspoon?”
I’m looking for an understanding of how conversion tracking works – ideally within a large analytics package, like GA or Omniture.
“According to Comscore, Company XXXX seems to been outpacing our growth in search over the past 12 months, why do you think that’s the case?”
I’m looking for answers that utilize some of the paid or unpaid tools on the market. I’d expect someone to recommend doing a comparative link evaluation using tools from Majestic, SeoMOZ, Raven or even something free like Blekko.
This is a great time to let the candidate actually log on to a tool and walk you through how they use it. Having a Majestic account is very different from being able to use one; in the same way that having a gym membership has not slowed the progression of my increasingly soft midsection.
“Tell me about Caffeine or May Day.”
I’m not looking for my favorite beverage or descriptions of young girls dancing with ribbons. These are two named algo updates from 2009/2010 and familiarity with them indicates some duration of hands-on search. If you are looking for someone with serious longevity, ask about the Google Dance.
“Here is an excel file with data for all of the 600,000 restaurants in the United States from Urbanspoon. Turn this file into a list of cities with more than 1,000 restaurants.”
The answer involves pivot tables and should take about 1-3 minutes.
“Now, imagine we purchase a list of restaurant emails and want to append our huge database with those emails … how would we go about doing this?”
The answer involves a VLookup, which can be persnickety and tedious – fair to assume if they know the answer, they can work through the actual implementation.
“We’re testing television advertising to evaluate our ability to extend our brand from a directory to a reservations destination. How would you evaluate the ROI of this?”
This question requires a fairly complex answer and I’m looking for nuances that include:
- What is your business objective?
- How is the test set up . . . what is the control I’m comparing this to?
- Some concept around lift and then extended lift (this is more of a branding answer, but ties into the notion of multiple touch points instead of focusing entirely on last click attribution – which is a very common mindset.)
- Search bonus for looking for lift uptick in branded search as well as direct traffic.
“How can we rank better for ‘Romantic Italian Restaurants in Redmond’ – here’s the page, but it ranks on the second page for that term.” (Assume said page has standard on page optimization in place.)
This is a red herring question - what I really want is for someone to tell me something along the lines of:
- “ranking reports are a waste of time and here’s why”
- “you should look at the inbound search traffic instead of the ranking report for a specific term”
- “this is a long tail term, if you spend time chasing every long tail term, you’ll miss the forest for the trees.”
At some point, I’m going to dig really deep on something where I have a lot of experience (say local or news search) where I know the candidate is going to hit the ceiling of her knowledge.
In this case, the right answer is, “I don’t know.” There’s a lot of changes in search, making it impossible to know everything. Being able to tell your interviewer (or boss) “I don’t know” is very important. There are already enough people trying to BS their way through the search industry and I don’t want them working for me.
Conversely, (and just as importantly) I want the candidate to enlighten me. A great candidate should be able to teach me something.
If all else fails to draw this out, try: “Tell me something I don’t know about search.” The close cousin of the tell me something question is forward looking: ”what is the next big thing we should be thinking about?” or “My CEO is all atwitter about Pintrest, how should this play into our search strategy?”
“Tell me what you know about me.”
This isn’t a narcissistic probing into my favorite topic; but I’m really asking to see if a) the candidate has done any preparation and b) if the candidate is social. The integration of social into search makes this question increasingly relevant, especially for smaller entrepreneurial companies that may not have a “social” department or dedicated function.
Smart candidates avoid this question by dropping hints about the interviewer’s background during the interview: “I saw you played rugby”, “lived in Ireland”, “have kids” to let the interviewer know they’ve done some research. This can also take the form of, “In the New York times article last week about Urbanspoon…”
Finally, the best question in any interview is a series of “whys”. I interrupt repeatedly with why.
Why? Because it challenges statements. Why? Because some candidates try to gloss over hard questions with assumptions. Why? Frequently, I find explaining assumptions leads to discussions on fascinating tangents. Why? Because their experience is very different to mine and I want to know what they’ll bring to the table.
Onerous interview process? Uncomfortable? Maybe. But, I’d much rather have a tough hour long interview than a tough month or two or six with a poor fit. And let’s be honest, anyone worth his or her salt is going to have to answer much harder, on the spot questions across the organization every week.
For more on hiring awesome SEOs, check out Luc Levesque’s post: How to Sniff out Rockstar SEO Talent.
Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.