If you’re looking for some joy from your AdWords accounts this holiday season and beyond, Google’s new “Automated Rules” tool, which must certainly be in the running for the most-dangerous AdWords feature ever released, might only makes things worse. Automated Rules allows AdWords users to specify monetary amounts (or percentages) by which to raise or lower bids or budget limits whenever certain conditions are met. The example Google gives is if your average position is “worse than” 4.0, Automated Rules can increase your bid by, say, 20%. But like the dad in the film, Google’s Automated Rules allows users to do what they want, but gives very little guidance on how to do so in lower-risk ways that maximize return.

One thing that’s always bothered me is whose fault is it, anyway, that Ralphie hurt himself? Sure, all of the adults warned him he’d shoot his eye out, but no one instructed him how to use the gun properly, now did they? The closest was when his mother says to not shoot at “any animals or birds.” The dad gives him the gun, but not safety glasses. He helps Ralphie load the rifle, but doesn’t say to not shoot at hard surfaces nor to be mindful of his stance and to not put his finger on the trigger until he’s ready to shoot. Even manufacturers of power tools for grown adults write instruction books and put warnings on their products to tell people how to use them properly. But by giving advertisers the ability to set rules without instructions for doing so, it’s almost as if Google is trying to create a problem.

For example, imagine that there’s an isolated keyword whose profit-maximizing bid is $4 (at which point your CTR will be 6%). Higher bids will tend to result in higher position (and thus a higher CTR) and lower bids in lower position (and thus a lower CTR). Your current bid is $3.80. Let’s say that you use Automated Rules to increase the bid by 20% whenever your CTR is lower than 6% (that is, whenever your bid is less than $4) and to lower it 20% whenever your CTR is higher than 6%. Because the ad doesn’t get an enormous number of impressions per day, you decide to only run the rule once per week.

The first week, because your bid is too low, your CTR will tend to be low, so Automated Rules will raise the bid 20%, from $3.80 to $4.56. Your CTR will now tend to be too high, so your rule will drop the bid by 20%, to $3.65. You’re now actually further from the optimal bid than you were when you started!

Using this method, it will take your rule 8 weeks for the bid to reach within 5 cents of the optimum bid. Having the rule change the bid less aggressively (by 10% or 15% per week) would take even longer to reach within 5 cents of the optimal bid than changing by only 20%. Interestingly, changing the bid more aggressively (by 25% or 30% per week) would also take longer! And changing the bid around 39% per week results in an infinitely repeating pattern of bids that never reaches the optimal level.

Shouldn’t Google at least warn you when you specify a rule that has this possible behavior?

Part of the problem with AdWords Automated Rules is the lack of built-in guidance for the selection of rule conditions. Even Google’s “help page” describes only the mechanics of specifying rules, but says nothing about how to make rules function well and avoid conditions like the one I described above.

In addition to leaving out warnings, Google has also left out critical metrics like the incremental cost-per-click (ICC). In a YouTube video from 2009, Google’s chief economist Dr. Hal Varian showed how to use Google’s Bid Simulator (GBS) to calculate what he called “optimal or near-optimal bids” by finding the bid at which the ICC is equal to your value-per-click (VPC). So why doesn’t Automated Rules allow me to input my VPC and have Google use Dr. Varian’s method to calculate the optimal bid? Who in their right mind would let a rule run for months to change a bid from $3.80 to $4.00 when they could just use Dr. Varian’s method to calculate the optimal bid on the very first day?

Another problem with Automated Rules, besides the important functionality that has been left out, is the dangerous functionality that is present, like letting managers use their ads’ average position, Quality Score or CPA to change bids.

Google’s example used average position to determine when bid changes are made. But this is unwise. Say that you have an ad that’s in position 5 but you want it in position 1. You implement a rule to be run each day that says: “If average position is lower than 2.0, raise bid by $0.50.” At noon on the first day, this rule raises your bid from $0.50 to $1.00 (because your current position is 5). Your average position gets reported to you the next day as 3.0, so the rule raises the bid again to $1.50.

But what if your average position was 3.0 because you got 10 impressions in position 5 in the morning and then 10 more impressions in position 1 in the afternoon? That is, the $1.00 bid put you in position 1, so there was no need for the rule to raise your bid again. Simply because of the way you have set up the rule, you will ensure that you are overbidding for the top position.

Further, Google lets managers use their keywords’ Quality Score, but Google has confirmed several incidents in recent months where Quality Scores were simply being reported incorrectly and it is not certain that these issues have been fully resolved. Even when Google’s reporting does work correctly, Quality Scores can still change dramatically for no reason.

And Automated Rules lets managers use their cost-per-acquisition (CPA) to guide bid changes. But in a recent post Braking the Rules, I showed that, for a set of keywords which have a 5% conversion rate, using “one-half of your target CPA” and “2X your target CPA” as the conditions that prompt a bid change will cause you to change your bid unjustifiably, on average, 53% of the time.

To many (if not most) AdWords users, it is not obvious that these position, Quality Score or CPA-based rules might generate problematic behavior, but yet Google has built no safeguards into AdWords to prevent this from happening nor to warn you before it does.

Don’t get me wrong: rules-based bidding has its place in a PPC marketers toolkit, especially with newly added words for which there is little or no performance data. But managing the daily bid changes for most of your account is generally not one of them. Fail to heed that lesson and, well, you’ll shoot your eye out, kid.

“The clickthrough rate (CTR) was 2% on Friday, but 3% on Saturday. That’s a 50% increase. 50% is a lot, right?,” they ask. Well, it certainly is for income tax rates, but not necessarily for differences in clickthroughs. What if Friday saw 2 clicks from 100 impressions and Saturday saw 3 clicks from 100 impressions? Doesn’t sound so impressive anymore, does it?

Part of the problem is that it’s simply impossible to tell from that few impressions whether both have an inherent CTR of 2.5% (and you just happened to see 2 clicks for one and 3 clicks for the other) or whether they legitimately have different underlying CTRs.

Imagine a more extreme case: one ad has a CTR of 2% and one has a CTR of 100%. We see four impressions and all get clicks. How likely is this to be the data for the 2%-CTR ad?

Well, if it is the 2%-CTR ad’s data, then there’s a 2% chance that the first impression would generate a click. That’s about the same as the chance of randomly drawing the ace of spades from a well-shuffled deck of cards. And there’s a 2% chance that the next impression will generate a click, which is about the same as reshuffling that deck and then randomly drawing the ace of spades again (without any sleight-of-hand trickery).

So, the chance of seeing 4 clicks from 4 impressions for a 2%-CTR word must be very, very small, but (please take a minute to convince yourself of this, if you need to) it’s not absolutely zero. Even an ad with only a 2% CTR still might possibly generate 4 clicks from 4 impressions. It’s improbable, but not impossible.

That is why statisticians rarely seem to give a straight answer to whether two ad’s CTRs are different or not. “Statistical significance” is not really a Yes or No situation, it’s just the probability of seeing a certain sequence of events (like four ace-of-spades in a row) not happen purely by chance. Every new impression increases the certainty in our answer, but there is no specific amount of information that seals the deal.

By convention, statisticians often set an arbitrary cut-off of “5% chance of being explained purely by randomness” for classifying whether or not a difference is “statistically significant” or not. That’s why when a magician declares that he’ll pull a certain card from a deck, and then actually does so, the average geek in your life will joyously exclaim, “That’s statistically improbable!” We know that there’s less than the 5% cut-off chance that that card appeared purely by luck.

Imagine now that we have two ads, Ad A, for which we have observed a 2% CTR, and Ad B, whose observed CTR is shown on the x-axis of the graph below. The graph shows the number of impressions (per ad) we must see to be 95% certain that the two ads have different CTRs.

If Ad A has seen 2 clicks from 100 impressions (2% CTR) and B has seen 14 clicks from 100 impressions (14% CTR), then we can be more than 95% certain that Ad B’s CTR is higher than A’s. If the observed CTR of Ad B is only 3%, then we actually need nearly 4000 impressions each to be 95% certain that Ad B performs better. That’s why the difference in observed CTRs between the Friday and Saturday ad performance wouldn’t look so impressive if they only had 200 impressions between them.

As the CTR of Ad B approaches 2%, it takes staggeringly more and more data to differentiate the two ads. Trying to tell a 2.00% CTR ad from one with a CTR of 1.95% (or 2.05%) takes more than a million impressions each. And, if the two ads perform identically, with exactly a 2% CTR, obviously even an infinite amount of data couldn’t tell them apart.

Though the concepts I’ve described above are (hopefully) now very clear, unfortunately some of the web-based tools for differentiating CTRs seem to have disregarded them completely.

For example, if one ad got 1 click with a 25% CTR (that is, 4 impressions) and a second ad got 2 clicks with a 100% CTR (that is, 2 impressions), Splittester.com by Brian Teasley and Perry Marshall says: “You are approximately 99% confident that the ads will have different long term response rates.” 99% confident from just 6 impressions?! No, I’m not. If I flip a coin 4 times and get 1 “heads” and another coin 2 times and get 2 “heads,” I wouldn’t be 99% certain that either one of their per-flip chances deviate from 50% at all.

Supersplittester.com, a similar site by Dr. Glenn Livingston (I presume), has similar deficiencies. For the case of Ad A, with 4 impressions, 1 click (25% CTR) and 1 conversion (100% CR), and Ad B with 2 impressions, 2 clicks (100% CTR) and 1 conversion (50% CR), the site tells me both that “Ad B has a higher CTR than ad A (99% Confidence Level)” and that “Ad A has a higher conversion rate than ad B (80% Confidence Level).”

Frankly, the only thing I have 99% confidence about is that Teasley, Marshall and Livingston should have a second look at their computer code to see what’s going wrong.

In the McKinsey Quarterly, Google’s chief economist Dr. Hal Varian said: “I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s?”.

He’s absolutely right. In 1990, only a handful of geeks knew what a “homepage” or an “email” were. Ten years later, few people didn’t know. Likewise for search marketing, even basic concepts like determining a confidence interval to identify statistical significance can still seem esoteric. But the industry is quickly realizing that being able to do these calculations is not just for geeks anymore.

They even run ads on other search engines. Search for “Google” on Yahoo and you might see:

(The 3-line description is a result of the way some ads display in Yahoo’s results.)

Various blog posts have offered advice on writing “killer” ad copy, suggesting (among other things) to use the keyword in title of the ad and to also use the keyword in the ad’s description text. Others advise including a strong call-to-action, though the jury still seems to be out on the subjects of best use of capital letters and on whether or not to use an exclamation point. Google itself suggests to test multiple ads in each group.

Checklists are for the grocery store

The reason why search engine marketing can often be frustrating perhaps is already obvious: even for Google, “industry standard” practices can’t seem to be universally applied. The two ads above both use “Google” in the title and description, but one includes an exclamation mark while the other does not. One treats “Homepage” as a single capitalized word while the other treats it as two words (but strangely doesn’t capitalize the word “page”). And neither seems to be part of an ad group that has multiple ads, since these are the only creatives I have ever seen for this search query. If you use a checklist of “best practices” when writing ad creatives, at least be aware that Google’s ad writers don’t seem to do the same.

It might be that Google simply isn’t trying to optimize the performance of their ads on their own website (since they don’t change their own bottom line by charging themselves for clicks). But it seems harder for me to believe that Google isn’t trying to optimize the ads they place on Yahoo and Bing. Even if money isn’t an issue, certainly setting a good example and maintaining the appearance of competence are.

So, I decided to examine Google ads on Yahoo and Bing further to see what lessons we can learn from Google about writing good ad creatives:

Lesson #1: Put all, or most, or some of the keywords in your ad

Yahoo and Bing don’t seem to run Google ads for terms specific to Gmail (Google’s email service), Google Maps, or Chrome (their web browser), but they both run ads for Google’s new Nexus One phone. For the query “nexus one phone,” Bing and Yahoo both run the same 3 ads:

Notice that none of the ads mentions the full keyword in the title, even though there is enough space in the titles of the top two ads to say “Nexus One Phone by Google”, instead of just “Nexus One by Google.” And, none mention the keyword at all in the text description.

Lesson #2: Emphasize benefits, not features (or the other way around)

A “golden rule” of writing good PPC ads is to emphasize benefits, not features. So, for example, you should say that the machine offers “fast performance” rather than a “1 GHz processor.” But looking at the three ads above, it appears that all emphasize features (“4 GB Flash Memory,” “1 GHz Processor,” “Tech Specs”), not benefits. Perhaps this “rule” of killer PPC ads should be revamped to say: “Emphasize benefits, not features… except when you should emphasize features, not benefits.”

Lesson #3: About attention-grabbing headline

Another common piece of advice is to write an attention-grabbing headline. Bing doesn’t seem to run Google ads on terms related to finance, but for the queries “stock quotes” and “financial information,” Yahoo shows:

“About Stock Market?” “Financial information?” I don’t know if I could think of a headline less enticing than those. Maybe, “About information?” Using bland headlines like these might make sense for testing purposes if Google had similar ads in the rotation which had much more enticing headlines. However, in apparent violation of their own advice to test multiple ads in each group, these seem to be the only ads they show for “stock quotes,” “stock quote,” “stock market quotes,” “financial information” and similar queries. Notice also that few words are capitalized in these ads, not even the display URL, even though some suggest to capitalize the display URL. And as far as the advice of using words that evoke emotions goes, these ads are stone-faced. So, perhaps Google has already done their testing and discovered that these lame-looking ads get the best response—in which case, maybe writing enticing titles only attracts poor-quality visitors.

Lesson #4: Keep the ad and the landing page closely related to the keyword (unless you have a new webphone on the market)

Interestingly, the ad which appears on Google itself for “stock quote,” “stock quotes” and “stock market quotes” is:

It seems that rather than take you to their highly relevant Google Finance page, as their Yahoo ads do, they’d rather use your “stock quote” or “financial information” query to show a less-relevant ad which cross-markets their shiny new cell phone. (I wonder what the landing page quality of their phone page is compared to their finance page for this term. For that matter, I wonder what Quality Score a low-relevance ad like this one has…)

Lesson #5: Always-be-testing everything but the display URL

Stock quotes and webphones are one thing, but Google makes most of its revenue from selling pay-per-click ad space, so let’s get to the heart of the matter. On the query “pay per click advertising,” they run at least six different ad creatives on Yahoo and seven on Bing, five of which are common to both search engines. Of the five ad creatives in common two use “pay per click advertising” only in the title, one uses these words only in the description text, one uses these words in both the title and the description, and one uses them in neither the title nor the description text. Here’s how the ads appear on Bing:

These ads on Yahoo are the same, but with periods in place of the exclamation marks. So, it appears that Google might be doing full factorial testing of the titles and text on some terms that are core to its business.

Though some suggest to test multiple versions of your display URL, it seems that Google has disregarded this advice, since the display URL on all the variants of the “pay per click advertising” ads on both Yahoo and Bing are identical: www.Google.com/AdWords.

Lesson #6: Don’t sweat the small details

Surprisingly, in one of the ad variants that appears on Bing but not on Yahoo, Google’s ad has a minor formatting error that has unwittingly tripped up many search marketers:

By not realizing that when ads appear in the “promoted positions” above the natural results, the description text is presented on one line, Google’s ad writers left out a blank space between “Minutes” and “Instant.” Try to not let this happen in your ads!

Lesson #7: Sweat the small details

When testing ad variants, it appears that Google rotates ads evenly on Yahoo and Bing, but uses the “Optimize” feature (which delivers ads with higher clickthrough rates more often than those with lower CTRs) for ads which are shown on Google itself. For example, for the query “search marketing,” Google has at least seven variants on Bing (some of which use the keyword in the title, some which do not, etc. etc.), but on Google itself, they show only two ads, the first of which appears about 90% of the time and the second only about 10% of the time:

Personally, I can’t see anything that would explain the apparently large difference in CTR between these two ads which would cause the first to be shown so much more often, except perhaps that “get” comes from German (“gessen,” by way of Old Norse “geta”), while “gain” is from French, and English words of German origin are often perceived as more “visceral” than their French or Latin equivalents. So, perhaps that mere fact explains their apparently wildly differing CTR’s and very small details can cause enormous differences in behavior.

Lesson #8: Use Google’s new boldfacing algorithm to your competitor’s advantage

As a final note, Google has recently begun boldfacing text for words which are synonyms for, but not identical to, words in the query. This might expand the range of words creative editors would consider using in the title and text of their ads. But, the differences between which words Google boldfaces and which is does not can be interesting. For example, for the query “free web mail” Google seems to run at least three different ads, two of which mention Gmail (one is shown below at left). For the query “phone apps,” they seem to have at least six different ads, one of which mentions the iPhone and is shown below at right:

It’s revealing that Google’s boldfacing algorithm considers Apple’s brand term “iPhone” (a competitor to Google’s phone) to be a synonym for “phone,” but Google does not consider its own term “Gmail” to be a synonym for “mail”.

The general rule for people who write PPC ads seems to be: there are no general rules. Forget about “best practice” checklists because they don’t even seem to work for Google. Instead, just test as many ad variants as possible, even ones that you wouldn’t think might work well.

Lesson 1: Being timid is like sending your competitors a fruit basket

Since each ad position, as a rule of thumb, tends to get about 40% more traffic than the position below it, dropping several positions in the rankings, or dropping out of bidding entirely, results in multiple competitors getting substantial increases in traffic at the rate they were paying for their lower position. You wouldn’t send your competitors a fruit basket, so why give them higher ad positions at a discounted rate? One reason that might spring to mind for many account managers is “Because it saves me money,too!” An account manager I will call “Mike” once asked me something like, “If a word can get 100 clicks a week for $2 each, why not instead reduce its bid and get those same 100 clicks spread out over a month for $1 each?”

It’s a very good question, since Mike’s method sounds like it would get the same click volume for less cost, but there are two key times when it can be problematic: (1) when you don’t have a monthly budget limit, and (2) when you do. Let’s consider the first case first.

Lesson 2: There is only one “bidding strategy”

Some PPC marketers talk about “bidding strategies” the way they talk about market strategies and business models, as though there are many of them available and the marketer’s job is choose which is most appropriate and most likely to succeed in a given situation. In their book Search Engine Marketing, Inc., Moran and Hunt provide a table of what they claim are many different strategies, but as Google’s chief economist Dr. Hal Varian’s YouTube video “Google AdWords Bidding Tutorial” shows us, there is only one: For each keyword at each time, bid optimally.

Consider a keyword (called “A”) which has a 5% conversion rate and makes $300 in revenue per conversion. A second keyword, “B” is identical to A except that it has only a 2% conversion rate. The monthly performance figures shown in the table below for these keywords are fictitious, but realistic.

We can see from the table what a striking difference the lower conversion rate makes in both the monthly profitability of the words and in the range of bids over which a word is profitable, which is why people like Tim Ash, author of Landing Page Optimization: The Definitive Guide To Testing and Tuning for Conversions, are such vocal advocates of conversion optimization. According to Dr. Varian’s method, if we had no budget limit, we could find our best bid for each keyword by just selecting from the table the bid with the highest profit (which for keyword A would be $7.20, bringing a profit of $2,443 per month and for keyword B would be $2.40, bringing $719 per month). These are the only optimal bids for these words, since our economic goal as advertisers is to maximize the amount of profit we make per heartbeat.

In that rare utopia where an advertiser has an essentially unlimited budget, AdWords acts as what I call “the bank of Google.” I have one client in particular who, despite the economic downturn of the past year, can’t spend money on pay-per-click (PPC) advertising fast enough, because for each dollar they spend they ultimately generate on average $2 in profit. For them, AdWords is like a bank where for every $1 deposited, their account is credited $2 by the end of the day. In fact, in “Online Ad Auctions” (The American Economic Review, Jan. 2009) Dr. Varian has shown that the ratio of value delivered to advertisers divided by the revenue to the auctioneer (that is, Google) should be about 2 to 1, at minimum.

So when the budget is essentially unlimited, for Mike the account manager to overbid on any keyword reduces profit by wasting money and to underbid reduces profit by wasting time. He has only one optimal bid for each word and should use that bid until the performance estimates on the table change.

Lesson 3: Sometimes a pawn is your strongest piece

Done properly, PPC marketing is a measurable profit center, even though some see advertising as just an expense and therefore put artificial limits on spending levels. So, if we were not permitted to spend more than, say, $500 per month, we could put it all into keyword A (the more-profitable keyword), bidding $4.60 per click for a profit per month of $2,380, which is far more than we could make on keyword B. The question is: How much of our $500 budget should we consider shifting from the more-profitable keyword (A) to the less-profitable keyword (B)?

At first glance that question sounds absurd. If we have a limited budget, why not spend it all on keyword A, which has a higher conversion rate and generates two to five times as much profit at any bid as keyword B? Well, because it so happens that there is a better way to spend $500 than by putting it all into keyword A. If we were to bid $3.20 on keyword A and $1.70 on keyword B, then we will make $2263.34 + $706.80 = $2,970.14 (which is far more than bidding solely on keyword A) and still meet our budget limit. You can check the math for yourself.

The lesson for online marketers is that bidding optimally on multiple keywords simultaneously under a constrained budget is not simply a matter of reducing our bids until some desired spending level is attained, as Mike suggests, nor just selecting the largest profit from a table, as Dr. Varian suggests for the case of an unlimited budget. It requires rigorous calculation to avoid missing available profit, and there is no simple shortcut except to be aware of the pitfalls of budget-constrained bidding and to avoid them.