The SEM RFP: Questions You Must Ask

Picking the right search engine marketing (SEM) firm is hard. The quality of service and performance varies wildly, but the marketing materials often make it difficult to separate the experts from the pretenders.

Strangely the process is much more akin to hiring a law firm or an accounting firm than it is to hiring an ad agency. More like hiring an engineering firm than an architect; A football coaching staff rather than a band director; an historian rather than a novelist. It’s not that the job doesn’t demand creativity, it’s that the creativity must be exercised within some very well understood limitations of reality and what actually works.

Where ad agencies might have radically different ideas about how to market your brand — a talking gecko to sell insurance, for example — search marketing is far more about algorithms and execution of well understood principles. You want to hire the firm that has a track record of success, a sterling reputation and a fair fee structure.

Here are some cut and dry questions that slice through the malarkey of sales pitches and give a real glimpse into how each firm you’re considering approaches search.

• General business
  1. What fraction of your clients are in my vertical? (retail, financial services, travel, etc)
  2. What fraction of your employees work in client services vs marketing? This can be highly revealing. Firms with a large ratio of client service staff to marketing have built their businesses on retaining clients. Firms with a lower ratio may provide less service, or expect more client attrition, or are seeking rapid growth. The latter can compromise quality, as hiring and training quality analysts takes time. As a benchmark, our ratio is 10 to 1, but we’re nuts. 5 to 1 is probably still okay.
  3. How much money does your tenth largest client spend per month? How many keywords comprise your tenth largest keyword list? Size isn’t everything, but this will help get a sense of whether your account will be among their biggest, or no where close. If their tenth largest client is tiny, that’s a bad sign. If their tenth largest client is several orders of magnitude larger than you, that might indicate that you’re not going to be a very big fish in this pond which can be reassuring, but might mean you’re ignored.
  4. What types of clients fit best with your agency and what types of clients aren’t right for your agency?
  5. How do you rank the following elements of paid search in importance?
     • Account structure
     • Ad copy
     • Bidding
     • Data analysis
     • Keywords
     • Landing pages
     • Match-types and negatives
     • Quality score

     RKG has strong opinions about how these should be ranked for most advertisers to generate the greatest ROI, but that may not be your measure of success. If you don’t like their ranking then you may not like their approach to search.

• Keyword construction
  1. Does your agency build and maintain the keyword list or is that your clients’ responsibility? Almost every agency will say they build and maintain the list, but many don’t do it well and only maintain the list if goaded, so a couple follow up questions are needed.
  2. How do you build the list initially? If they talk about software packages like Hitwise, or their own trademarked “solution,” be concerned. Machines do a lousy job of keyword creation. Automated processes for building out product/SKU level variations is essential, but the core of the list should be generated by a human controlled process
  3. How much keyword list maintenance is required on an account like ours? A good answer here will depend a great deal on your business. A retailer with huge product turnover A really good answer might conclude with “…but we’ll see. At some point we may find that keyword additions and maintenance doesn’t add much value and we should focus on other things then.”
• Bidding: Many firms claim to have the most advanced bidding system, and will wear out their thesaurus describing their “cutting-edge,” “world class,” “enterprise level” systems. Below are some questions that should cut through the bluster.
  1. Who built your bidding system? You might find out that the “world class technology” they tout is in fact rented from some other company. That’s a bad sign, indicating that this agency has outsourced one of the two most important pieces of their service offerings. They might be okay, but by definition, they’re not leaders. If they built their own system, do the architects have the kind of statistics credentials needed for building a top-notch system?
  2. Is the foundation of the system built on finding the right position on the page, or bidding what the advertiser can afford to pay for traffic on each term? Don’t let them get away with “both.” Either they’re letting your competition determine the bids or they’re letting your economics determine the bids. We believe in the latter, but if you’re looking for an agency to do position bidding, make sure that’s what they do.
  3. How granular is the bidding: ad group level, keyword level, or ad level? For a major program, you need the most granular bidding, ad level, to differentiate between multiple versions of keywords that might be running on different match types, different syndication settings or different geographies.
  4. How does the system handle low traffic ads? Don’t expect them to share their algorithm, but they aught to be able to explain the concept without giving away trade secrets.
  5. How does it handle seasonal effects, short term promotions (10% off cashmere sweaters this week only!) etc? A bad answer will sound like “oh, our algorithms recognize the slightest changes in performance and react accordingly.” Sounds great, but what it means is they’re focused far too much on the most recent results and given the spiky nature of PPC the result will be whipsaw bidding and poor performance.
• Data analysis
  1. Can your agency report on performance data rolled up in different ways, eg: by product sub-category, or by manufacturer, or by the type of landing page, or by other ad hoc rules?
  2. Could your team pull that kind of report together in an hour? Every agency will say “yes” to the first question, but many will do so with trepidation. Cross-cutting views of data requires flexible data structures to do easily. If they don’t have flexible data structures, if the only data structure available is the account structure — which isn’t flexible at all — cross dimensional analysis is incredibly painful and slow. The point isn’t that you get the report you want faster, the point is that if this kind of analysis is hard for them to do, you can bet your last dollar they don’t do this type of analysis on their own, nor can their bidding system be terribly advanced.
  3. Who would do this type of analysis: my daily contact, or some other team? If the answer is “a separate team” that’s bad. Doing data analysis in the absence of detailed account knowledge causes problems. Those doing the analysis don’t know that you were pushing certain brands because of manufacturer discounts or co-op dollars, or that there was a huge promo two weeks ago on ski jackets, or whatever. Not knowing the details makes analysis less valuable and potentially dangerous.
• The reference calls: Almost any agency can find two or three happy clients to serve as references. Your job is to figure out whether they’re happy for the right reasons, or whether they just don’t know any better.
  1. Ask the happy client: “What criteria do you use to evaluate your search vendor?” You may find that they judge their vendor strictly by how well and quickly the vendor completes task lists, or how slick their reporting interface is. If they don’t talk about keyword level performance, data analysis, etc, that may mean that those critical differentiators aren’t what this particular advertiser cares about.
  2. Other than this agency, which others have you worked with? An answer of “none” isn’t necessarily bad, but those who can compare and contrast agencies often have more valuable perspectives to share.
  3. Do the client references get search? Do they think about search the way you do? The highest compliment to an agency is if clients who really understand the game are pleased with the service they get.

  Don’t be afraid to ask for more references with specific attributes — I’d like to speak with an internet pure-play selling low-margin products who’s focused on ROI, er whatever — that will allow you to speak with folks who think like you. If they struggle to find a client who fits your criteria it may mean they don’t have any — maybe that’s okay with you, maybe not — or worse, they have them but don’t want you to speak to them!

As the SEM space matures, the quality of management firms should improve to the point that you can’t go far wrong. Reputations and track records will be more widely known, third party rankings will become somewhat meaningful and the cream will rise to the top. That process is in its early stages, so be vigilant, ask hard questions and use the answers to separate wheat from chaff.