• Steve Cameron

    Great article.

    QS is the “holistic” benchmark for campaigns – it is a reflection of the keywords, ad copy and landing pages – it’s hard to affect QS by focusing on only one of these.

    But – whilst optimizing for higher QS is sensible, the truth is that optimizing for a better user experience and, in particular, higher relevance, will affect QS positively in any case, and these are, perhaps goals of a higher order.

    At the end of the day Google has never really wavered from the relevance mantra – and QS is their way of guiding us back to the straight and narrow by effectively punishing us for straying.

  • Larry Kim

    hey thanks for this note. i can publish this data in a follow up article. the challenge is mostly just time as opposed to some nefarious plot. believe it or not it is really hard to exclude brand campaigns from the analysis since that’s a super manual task in determining which keywords are the brand terms, then manually excluding them. i’m not a full time columnist here (i actually have a day-job running a company lol). I filed this story at like 3AM in the morning today.

    At a minimum i thought you’d appreciate the fact that i didn’t base my conclusions off of looking at 1 account or something like that. There are thousands of campaigns in that analysis representing a boatload of spend. I haven’t seen anything like that yet. When you look at so many accounts, you get some companies that do brand keywords, others that don’t.

  • Larry Kim

    thanks steve.

  • tedives

    I suspect a lot of what’s in the data is, late-funnel terms are the ones with higher CTRs, and the client campaigns are perhaps organized somewhat according to the funnel (or at least brand/nonbrand). So late-funnel terms/campaigns will have a high CTR and QS, and will convert great, and will cost less.

    There’s no way to really turn an early funnel term into a late funnel
    term, that’s just sort of how the cards fall.

    You can certainly get your CTR up by writing a better ad, and that’s a worthy effort – but you run the risk of bringing in viewers that won’t convert if you worry about CTR/QS overly.

    In fact, a creative can be thought of as a valuable filter to, rather than *maximize* valuable clicks, to *minimize* non-valuable ones. I think the industry tends to focus a bit much on maximizing rather than this valuable screening function.

    Taken to the extreme for illustration sake…if you sell lawnmowers and your creative has a .5% CTR, and you change your creative to have the title “Britney Spears Lawnmower” just to get more clicks, and you get a 4% CTR, and your QS shoots up…you’re clearly worse off, not better off; those clickers are very much less likely to be buyers.

    Sometimes a low CTR, and low QS, is exactly what you need, to separate the wheat from the chaff!

  • Martin Röttgerding

    Hey Larry,

    Cool post! Still, even though there is a correlation between visible quality score and CPC, I don’t think it’s a good KPI. It can tell you a few things if put in context, but in the end quality score (especially _visible_ quality score) is just a byproduct and not a goal.

    Regarding the connection to CPA, I think that’s a step to far. CPA is CPC times conversion rate. CPC is connected to quality score, but conversion rate is not. If your quality score goes up, your CPC’s might fall, but it won’t make people buy from you. That’s basically the difference between AdWords’ ‘optimize for clicks’ (aimed at improving CTR and therefore QS) and ‘optimize for conversions’ (aimed at improving the product of CTR x conversion rate and therefore CPA).

    Ummm, and I’m not so sure about this king of the realm analogy. How do wildlings, the undead and dragons factor into that? Winter is coming …

  • Martin Röttgerding

    This makes sense, but there’s another side to filtering clicks.

    If we assume that QS is basically CTR then the ad auction is just AdRank = MaxCPC x CTR. If CTR doubles, we can cut CPC in half, meaning we get twice the clicks for half the price, meaning we get twice the clicks for the same money.

    Now with the Britney Spears Lawnmower: Lets say you could get eight times the visitors (by increasing CTR from 0.5% to 4%) at the same cost as before. Then if those visitors would still include the valuable traffic from before, plus some less qualified traffic, this would be a great bargain. You would get the additional traffic for free.

    That’s the downside of pre-qualifying your traffic. In a pay per click world this seems to make sense, but AdRank is actually about what you pay per impression.

  • tedives

    Great point….CTR relative to others as I’m sure you mean… unfortunately it’s hard to know the competition’s CTR and what the curve will look like.

    Might be interesting to take into account the system-wide CTR in these sorts of analyses as well somehow, i.e. turning CTR into a relative CTR, I think that’s been available now in Adwords for some time, right, as it had been in Bing previously.

    Good clarification Martin!

  • Ryan Bruss

    How did you calculate campaign quality score? Did you limit this to only exact match keywords? If you did not, then your numbers won’t be correct. Broad and Phrase match keyword quality scores do not represent the functional quality score used to determine CPC (for Adwords).

  • Adrian Huth

    Love this comment. This is exactly why having CTR as the main QS factor is actually the opposite of the relevance Google supposedly is providing users. They created a system where, unless branded keywords, the advertiser has to improve CTR by creating more generic or even misleading ad copy and punishes people who use ad copy as a filter to separate the wheat from the chaff as you mention.

  • Martin Röttgerding

    In the ad auction it’s just CTR (or rather estimated click-through probability, to be precise). However, using a value relative to others wouldn’t change anything.

    An advertiser’s relative CTR would be CTR divided by average CTR. Since everyone’s CTR would have to be divided by the same number, the outcome of the auction wouldn’t change. If the AdRanks are 4, 5, and 12 or 40, 50, and 120 – the positions stay the same and so do CPC’s.

  • LampyB

    Don’t get your panties in bunch, it’s the internet and everyone’s entitled to an opinion. If you disagree, it’s your choice! I’d just like to see you post more relevant information here. Do you have any data to back your harsh words?

    “QS optimization is only going to move you up 2 points typically, 3 points in rare circumstances. Those are undeniable facts which are not debatable.”

    Where’s your data to back this up? If you’re going to call someone out, stating they must “bring forward the analytic views and prove me wrong” then I’d expect you to do the same.

    Also, your assumption states that the only keyword terms with QS of 8,9,10 are branded. This isn’t true. There are MANY instances where QS will be 8,9,10 for non-branded terms…I see it every day. You’re obviously the man when it comes to PPC, I guess we should all just take your word for it?

  • http://www.LeadDiscovery.com/ Jerry Nordstrom

    Larry I really enjoy your data driven articles.

    I would like to see the number of conversions overlaid on your QS vs cost graph.

    What is success?
    # of sales, # of leads/prospects, Low CPA, Lifetime Customer Value, Initial revenue?

    Incentive drives PPC management styles.
    Each PPC marketer will have different short and long term incentives to meet the requirements of their client/employer. The more sophisticated and accurate a company’s tracking and reporting system is, the better the can make the PPC manager account for performance.

    If a business never measures lifetime value of a new customer, only the initial revenue then quantity will trump quality making QS less important. Who cares if the customer sticks… that’s Sales and CS’ fault.

    If the sale takes place AFTER the online conversion, once again the PPC manager may not be accountable for quality, but rather quantity.

    The highest quality score may not always be the best point in the Sales/Revenue bell curve.

    To effectively manage your PPC agency or employee, make sure you have the data necessary to cross check PPC performance. Then you can develop the right incentives for the manager and s/he will create a campaign model to achieve those goals.

  • Larry Kim

    hi – my view of the data is that this (a strategy involving trying to go after low CTR keywords) will absolutely not work. I did not see any evidence that low CTR/QS keywords convert any better than high CTR/QS keywords. However, the CPCs are *so much higher* on the low CTR/QS keywords and *so much lower* on high CTR/QS keywords that the CPAs we just too high on low CTR/QS keywords.

  • Larry Kim

    yes, exactly. on average, you’re better off pursuing HIGH ctr/QS for a portfolio of relevant keywords.

  • Larry Kim

    impression weighted average quality score.
    so say you have 3 keywords in an account, a, b and c, with quality scores of 1, 5 and 10, having 0, 20, and 50 impressions, respectively.

    the impression weighted quality score is (1*0) + (5*20) + (10*50)
    then all divided by 70, which is the total number of impressions.

  • Larry Kim

    thanks for this note. it is hard to get data on customer lifetime value, etc. from the adwords – a lot of that data is stored in other systems within a company. But i agree that this kind of analysis is essential!

  • Larry Kim

    Adwords Quality Score, like Joffrey Baratheon, is the king many PPC managers love to hate :)

  • Larry Kim

    i see no big differences in conversion rates based on quality score. thus optimizing for CTR/QS results in (on average) far lower CPC’s and CPAs.
    i’d love to see your data if you’re seeing something different.

  • Ryan Bruss

    Hi, Larry. Sorry for the confusion. I understand impression weighting. My question was how did you use only exact match keywords in your calculation? If you did not, then your numbers won’t be accurate.

  • Larry Kim

    i used all match types. can you explain why this won’t be accurate? seems like a constant here. as long as i’m measuring the same thing across campaigns (and provided that there are a crap ton of campaigns) the constant cancels out.

  • Martin Röttgerding

    Yes, that’s my point: Conversion rate and quality score aren’t connected. No doubt, higher QS goes with lower CPA (and higher ROI, and higher profit) but that’s all just because of the lower CPC’s.

  • Ryan Bruss

    the problem is that the quality score for “keyword 1” displayed in adwords is actually the quality score for [keyword 1]. so, when a user searches ‘keyword 1 abc’ or ‘keyword 1 xyz’ and they click on an ad served by “keyword 1” the quality score used to calculate ad rank and cpc is NOT represented by what is displayed in adwords. because of this you would essentially be assigning the quality score of [keyword 1] to all the impressions for “keyword 1” and keyword 1.

    this is not well documented by Google

  • Larry Kim

    yes i understand this to be the case. the question i’m asking here is do you believe the propensity for a keyword to be broad matched against something else to be so significantly different on average when looking at across millions of keywords and billions of impressions/clicks. I think the noise just cancels itself out in this situation.

  • Larry Kim

    YES! that’s what i’m saying too.
    thus, i think it’s crazy to not be optimizing an account for CTR/Quality Score (essentially same thing).

  • Ryan Bruss

    yes i do. I’ve worked on many accounts with many different organizations with wildly different exact, phrase, and broad match impression rations. There seems to be too many unknowns.

  • Ryan Bruss

    there is another factor that needs to be addressed. quality score only matters relative to the competitive set of advertisers. it can also be skewed by the nature of the query. general queries tend to have lower quality scores. in those situations a 4 or 5 may be the best you can hope for, but it doesn’t really matter because all the advertisers are in the same situation.

  • Larry Kim

    that’s my point! it’s the same for all.

  • Manuele Caddeo

    I share your thoughts about everything. When you have as objective lead generation or Sales and in game there are keyword whit high competiton, keep QS as high possible is the first thing i take care. If you have an 1% conversion rate means every 100 clic you convert, and there is much difference if you pay 0,50 cent for clic or 0,70-0,80$ !!!
    Hi, Manuele

  • Ryan Bruss

    what’s not the same is the relative share of exact, phrase, and broad match impressions

  • Jeff Ferguson

    Interesting research, however, there’s a semantic issue here: Quality Score isn’t a KPI.

    A KPI would be metric tied to the success of the business for which the campaign is run. The success of the campaign is not the campaign itself, but what it accomplishes for the business.

    Therefore, Quality Score, along with many of other metrics discussed here are a diagnostic metric, not a KPI.

    You could chase Quality Score all day while your business goes under because you’re focused on the wrong metric. This is a habit we all need to get out of with a quickness.

    These articles are great from an educational standpoint, but as someone who is on the front line with clients, it has long since “gotten old” to have the advertiser state they need more clicks, a higher CTR, or a better Quality Score, when in fact what they need is more sales, revenue, leads, etc.

    Just the same, thank you for the research.

  • Nozulu Mchumane Mpafane Solly

    “If your quality score goes up, your CPC’s might fall, but it won’t make people buy from you.”

    Good statement Martin, this is so true

  • AdwordsExperts.com

    I’m always amazed at the haters, it makes me question why the hell anyone would employ your opinions.

    The article is on point in regards to the relationship of QS to CPA. Some of you need to get back to your labs and work on your research.