So, wouldn’t you think that after more than a decade of search marketing and literally tens of billions of dollars in the U.S. alone in paid search ad revenue, that someone would come along and actually build a powerful keyword generation tool?

Yes, there have been attempts in the past, and there are some handy tools out on the market today, but the general consensus with most search marketers is that these platforms are not all-inclusive.

They are simply pieces that still need much more manual manipulation to build stand alone keyword lists and groups. For our own clients, my team and I manually aggregate multiple keyword tools in order to put together what we feel is the most comprehensive list available.

Well, there’s a new player in the market. Billing themselves as “the world’s most intelligent keyword and categorization platform”, the Chicago-based technology vendor, Zenya, feels they have cracked the puzzle.

I met their founder and CEO, Stephen Scarr, a little over a month ago at an industry event put on by Marin Software. Talk about passion! Stephen could barely contain his excitement for what he believes is the best keyword building tool on the market. (Note: he is also one of the brains behind Info.com which he co-founded in 1999.)

Zenya has spent the last three years developing a sophisticated system of categorization that maps searcher intent of more than 600 million categorized keywords and growing.

Zenya’s first dedicated team included taxonomists, subject-matter experts, and structured data analysts who collaborated to build an entirely new taxonomy from scratch, one designed to cater to the unique needs of online advertising. Today, Zenya’s categorization algorithm can process more than ten thousands keywords every second with a database of over 1.4 billion keywords.

“Overwhelming demand for a better way to identify productive keywords led us to develop Zenya,” Stephen avidly explains. “Since the inception of search engine marketing, advertisers have constantly struggled with the time-consuming and arduous process of keyword research. Using available keywords tools, marketers spend countless hours identifying relevant keywords to create Ad campaigns of tightly themed categories (topics).”

To assist digital marketers, info.com conceived Zenya to create the world’s first repository of categorized keywords, making the keyword building task that could last days or weeks, take just minutes.”

The folks at Zenya believe there are three core values that search marketers will discover with their product:

1. To Save Time. Rather than spending weeks creating keyword lists, marketers can download thousands of relevant keywords in seconds.
2. To Gain Control. Using the category tree or advanced filters, Zenya customers can constrain and focus their results to pinpoint relevant intent and connect with customers.
3. To Achieve Insight. Marketers can make sense of unstructured keyword data and categorize entire keyword portfolios into more than 250,000 categories.

Sound good?  Well, let’s take a closer look at the product.

Take a look at the filters available for the current search of men’s shoes. You can exclude different things such as adult terms and branded terms or add your own minus (negative) ones. You can filter by category, intent, language, etc.

One of the more interesting filters is user intent. Zenya’s advanced categorization allows users to zero in on the types of terms that demonstrate various levels of the purchase funnel.

Categorical analysis is also very crucial to proper paid search. Using the categorical filters, users can better groups their keywords and develop granular messaging. Check out the keyword and category matches for the query cellphone plans.

Zenya has a variable pricing plan based on a points system. Users can opt-in at various levels to meet their needs.

Needless to say, Zenya is new to market and definitely has room to grow. For example, currently available as a consulting service, another feature in development is an automated gap analysis. This enables marketers to map their keywords sets to Zenya’s taxonomy of more than 250,000 categories and then source additional keywords in these categories as method of efficient campaign expansion.

As the paid search industry grows, so do the opportunities for vendors to garner budgetary dollars. Zenya is another platform that search pros should do their homework on and see if it should be another tool to add to their SEM toolbox.

But no one’s really solved it.

Some folks have come close. Most of the top-tier SEM platforms have very robust reporting tools—but the general consensus with search marketers is that they only go so far.

“Sure, the SEM tools sometimes do a great job solving for common problems. But when I have to do something fast, or even the least bit custom, or if I feel like I can re-use the tool again, I make it up in Excel and keep it around for future-use.” says search pro Crosby Grant, Director of Advertising Services with Stone Temple Consulting.

At the core of the issue is that most search marketers would love to not have all of the extra steps involved with importing SEM data into Excel and then having to pivot and filter it before even being able to start the analysis. If a tool can go 80% or even 90% of the way, that’s great, but then it still means Excel is needed somewhere along the way so why not just do it all there?

“We want to put a bullet in pivot tables and Excel,” said Jordan Franklin, director of social strategy at Clickable. “There was a time when Excel was an efficiency tool for advertisers. Now, with the massive volume and diversity of data channels, marketers need more powerful and flexible tools to answer their burning questions.”

Clickable’s plan to deliver reporting and intelligence across many channels starts with its Widget Dashboard (now available in beta).

Part of Clickable’s third-generation technology platform, the Widget Dashboard visually captures and displays all search and social marketing data.

The Dashboard gives digital marketers and agencies an easy-to-use platform for tracking and modifying campaigns across Facebook, Google and Bing. Already connected to Clickable’s ad management tool, it also enables future data integration of additional channels across social, search and video, as well as proprietary customer data sources. Upcoming integrations include LinkedIn, Twitter and Foursquare.

Their vision is to empower marketers and agencies to demonstrate ROI and improve decision-making through visually persuasive dashboards and intelligence that spans online and even offline channels.

Not only search and social, but display, YouTube, television ratings data, and even customer CRM — providing a holistic view of the entire marketing funnel, from branding, engagement, direct response and ongoing customer engagement.

Other companies do have similar dashboards, but Clickable’s plan doesn’t stop there. They know that once all of the marketer’s data in in one place, the next evolution of the tool will begin to deliver new business intelligence not only for search marketers, but for all key stakeholders in the marketing value chain — from account executives to digital strategists to chief marketing officers.

“The ability to leverage big marketing data has created opportunities to build tools and platforms that can surface insights around demand, supply, consumer behavior, segmentation, positioning and targeting,” said Max Kalehoff, vice president of marketing.

Let’s take a look at some screenshots from the current Widget Dashboard beta as well as some soon to be released features. Even though this is just the first step in Clickable’s plan for the tool, it still has some very strong functions that marketers are already leveraging for their campaigns.

As the tool’s name suggests, the dashboard is based on widgets which are the building blocks of the visualization features. You can build them from any metrics and then manipulate them into different configurations based on the types of reports you want to see.

As you can see below, widgets can mix data tables and charts/graphs to build out any combinations that you would want to see.

Want metrics? No problem. Any fields from data coming into the system can be chosen in the widget building process.

You can make your own custom metrics as well from any combination of data fields in the system.

For example, you could create a new column called “Total Cost with Fees” to include any markup from tool charges, agency fees, etc. This way ,you’re always looking at the right CPA and other revenue or cost related metrics.

On the bottom left, the Widget Dashboard is reporting on Clickable’s proprietary metric, Brand Advocates, which “defines people who like more than one post of a brand on Facebook, and either share or reply to at least on piece of content posted by that brand in a specific time period.”

Cool stuff! Right now, the tool is a very fast and intuitive dashboard that is competitive with the reporting features of other platforms on market.

However, Clickable maintains that this is in fact just step one to aggregate all of your cross-channel marketing data in one place. Once this has been accomplished, the idea is to bring in more outside data to bring context and deep business intelligence to the dashboard to make it a true SEM analyst’s dream tool.

For paid search, getting even a surface level audit of your AdWords account can be fairly expensive. A consultant would have to access your account, pull your reports, check your settings, and evaluate your Quality Scorse, match type usage, etc. Then, putting together their findings in a report (along with links to more information) could easily take hours.

Luckily, search marketing software and services provider, WordStream, has a new tool that can not only find issues with your AdWords account, but is also free for every search marketer to use.

WordStream’s AdWords Performance Grader free evaluation service is a quick and easy way for search marketers to instantly identify where improvements could be made in their AdWords campaigns. It also provides benchmark comparisons to other accounts with budgets in the same range.

I asked Larry Kim, WordStream’s Founder and Chief Technology Officer, what was the inspiration behind building this free tool:

“First, we do account audits and assessments for both prospective and existing clients – for example, we have this Quality Score Toolkit that is basically just an Excel file that does a bunch of number crunching on data from your AdWords account. People liked that but we wanted to make it more automatic.  Second was that we found that advertisers are generally curious to see how they’re doing in paid search.  And lastly we found that there just wasn’t any way to do an instant audit of a PPC Account and compare the results against other accounts.”

The AdWords Performance Grader examines key campaign features including:

• Effective use of negative keywords to control spend
• Quality Score for text ads and the keywords targeted
• Click-through rates on ads
• Impression share for ads
• Long-tail keyword optimization
• Text ad optimization
• Landing page optimization
• PPC best practices

How does it work? Simply input your contact info and Google login credentials (Wordstream will not share or save your login) and the report is delivered to you in moments. Couldn’t be easier!

As always in Search Marketing Toolbox, we go under the hood. Let’s take a look at some of the information from an example report offered by the AdWords Performance Grader to see if it might be a good fit for you.

At the top of the report, you will get an initial summary of the findings along with Wordstream’s propriety score for your overall grade.

Something every search marketer spends a lot of time thinking about is Quality Score.

You can also get information on your Click Thru Rates…

…and your landing pages.

Finally, my favorite section of the report is a check on common SEM practices and how your account stacks up.

There’s more to the report than I was able to show here. Also, as you can see from the screenshots, there are links to more information for search marketers to take the insights they find and actually do something more with them.

I asked Larry what other features are in the works for upcoming releases. He said, “The most exciting thing on the road map for the AdWords Grader is enhanced grading. Now that we’ve collected a bunch of data, we can do even more interesting things with the AdWords Grader scoring algorithms, for example, by providing more granular segmentation by industry, like “Finance & Insurance”, etc.”

What a great, free tool. If you’re one of the select few search marketers whose accounts are doing so well that you don’t need any outside expertise, then I congratulate you. For the other 99.9% of us, WordStream’s new AdWords Performance Grader is a fast, free way to get an extra set of eyes on your account.

Most of the time, this is due to the search engines personalizing their results for each user. We also know that things such as time of day, browser settings, previous searches, and even social network connections can influence the Search Engine Results Pages (SERPs) to individual users—more so when logged in to the engine account but also when a user isn’t logged in at all.

Geolocation, however, is one of the major influencers to customized SERPs that affects both the logged in and anonymous user.  We have keyword scraping tools such as AdGooroo, The Search Monitor, and Adthena that aggregate search results over many different locations in order to best report on the listings from a holistic perspective.

But for the most part, the available tools require you to collect several days’ worth of data in order to get a statistically significant view.  What do you do when you want to just quickly see a SERP in another location?

There have been various platforms and workarounds before, but LiveSerps is a tool that hopes to solve this dilemma once and for all.

The Firefox and Internet Explorer toolbar was developed by The Search Monitor and lets SEM/ SEO marketers to view live SERPs from anywhere in the world to validate ad placements and organic results.

It currently supports top tier engines in forty countries and twenty U.S. markets. SEM pros can also choose to view their data from a fixed Web perspective or choose various mobile devices such as iPhones/iPads, Android & Blackberry.

I asked Lori Weiman, CEO of The Search Monitor how they came up with this idea.

We realized that while our clients have huge data needs across thousands of keywords, there are some times when a quick live search view on a small group of terms is needed.  The preview tools provided by the engines have a lot of limitations and so just aren’t meeting all the needs of the busy search marketer.  For example, these preview tools only allow 1 look-up at a time and only on their own engine which makes multiple checks tedious, plus the results have the words ‘preview’ all over them which is poor quality for screenshots and client presentations.  Since LiveSERP doesn’t have these limitations – multiple searches can be run at the same time, it remembers your last search so that you don’t have to re-enter all of your settings again, and it allows for multiple devices, geos, and engine views simultaneously.

Okay, enough jibber-jabber…let’s go under the hood!

After downloading and registering for a free account, this is the toolbar interface that you will  see in your browser:

As you can tell, it’s very intuitive and allows for a very simple way to choose the engines, countries, languages, and devices you would like shown.

For more options, click the settings “gear” symbol on the far right, and you can select a few ways to customize how you want the pages to be displayed.

Here’s an example of a search performed on the terms “ski jackets”, “snowboards”, and “snowboarding boots” on Yahoo in Atlanta.

You’ll use up your twenty-five free searches per month fairly quickly, because each SERP is considered its own “search”.

For example, if you were to search two keywords, on three engines, in two different cities, that would count as twelve total searches (2 x 3 x 2 = 12).

Users can receive new search credits in the following three ways:

• Liking their posts on Facebook – facebook.com/thesearchmonitor
• Following them on twitter and retweeting their tweets @searchmonitor
• Buy more searches at any time in bundles of 100 searches per bundle

So far, the user reaction has been very positive.

“Live SERP provides our marketing team a standard for accurately monitoring our SEO and SEM placements across key geographic markets by removing the influence of social signals and caching. As an added benefit, we’ve even found it useful in discovering affiliate and competitor geo-targeting strategies we wouldn’t otherwise see in real-time.” said Jean Elias, Director Online Marketing, for CareOne Services, Inc.

There you go…another great tool for your Search Marketing Toolbox.

AdGooroo is well known in the SEM industry for their SEM Insights and Natural Search Insights products that help marketers get a very accurate view of paid and organic listings on Search Engine Results Pages.

In full disclosure, I’ve been using AdGooroo since 2006 and I have found their level of innovation, technology, and service to be at the highest of levels. I didn’t use the Trademark Insight tool until a few years ago, but it’s just one many products they offer in their suite of integrated tools.

Why Are Trademarked Terms So Important To Monitor?

Think about it: in most paid search marketing accounts, the branded terms are usually the highest converting and most efficient traffic stream you can have. Not only could competitors literally be taking money out of your pocket, but they could also be driving up your costs.

If they’re bidding aggressively, you’re probably following suit to maintain good average positions—so if you can knock some of those folks off of those SERPS, then you won’t have to bid as high to get in the same spots.

When you also realize that your quality scores could go up because your click-through rates could improve, it becomes obvious how impactful trademark wrangling could be.

As AdGooroo notes in their product literature:

Most companies do nothing to stop competitors and affiliates from advertising on their brand terms because of the difficulty in identifying offending ads. AdGooroo’s 24/7 automated brand monitoring service solves this problem by identifying those advertisers who are bidding on or using your brand terms in their ad copy.

Notice they mention affiliates as well. One of the biggest offenders (and I know this from personal experience) to trademark infringement are affiliates.

It’s become commonplace for most major advertisers who run affiliate programs to not allow their partners to run paid search ads on their branded (or trademarked terms) so that they’re not bidding against themselves or losing sales that they could have garnered.

I’d like to give these folks the benefit of the doubt and hope that when they do break these policies that they do so in ignorance and not on purpose…but the negative results are the same whether it’s competitors or affiliates.

So How Does Trademark Insight Solve This Problem?

Well, if you’re not up to speed on the engine stance on this issue (check out, for example, the AdWords trademark enforcement policy), the gist is that in most cases, it’s up to you to file a report.

Understandably, Google can’t sift through every advertiser on their platform and know who has the authority to use these terms.

So, right now, if you wanted to police your trademark terms, you would have to constantly manually search on your terms, take screenshots, fill out the paperwork, etc. Trademark Insight smartly automates this process.

Now for my favorite part of Search Marketing Toolbox, we go under the hood!

Once you load up your keywords, Trademark Insights takes over and checks for your terms across eleven search engines and fifty different countries. This prevents some rather savvy folks from trying to “geo-spoof” (not geotarget in your company known HQ areas).

After a day or two, here’s the kind of report you will see:

And here’s the detailed listing of the incidents…

You get a list of all advertisers and ad servers:

You can even see infringers based on destination and display URL:

Here’s my absolute favorite part of the tool which to me makes this product a no-brainer. In a few easy steps, Trademark Insight generates the proper engine form with all of the necessary info to file a complaint as outlined by the particular engine’s trademark policy.

How nice is that?

And finally, you can setup a daily email report to check if an incident occurred was discovered the previous day without even having to log in to the platform. Talk about “set it and forget it”!

If you’re a search marketer who has trademarked terms in your account and you’re not tracking infringement, it’s definitely possible you’re missing out on a quick fix to performance improvement. With AdGooroo’s Trademark Insight tool, you can monitor and file engine complaints quickly and easily.

Good hunting!

In fact, one of the major draws for me to SEM was that it was a gold rush to new territory.

Appealing to my Gen X trailblazer sensibility was the fact that there weren’t a bunch of stodgy gray-hairs telling us how it was supposed to be as many of my peers were discovering in their established career paths. No hard rules. No classes offered. My textbooks were the blogs; my classrooms were lightly attended search conferences; my homework was trial and error on Overture and the newly launched AdWords. I was a keyboard cowboy.

Feel free to slap me if I ever start a sentence with: “In my day…”

Now, over a decade later, this SEM is a thriving segment of the advertising ecosystem. With $18 billion in the US this year, set to rise to over $30 billion in 2015, the search marketing industry (with most of the dollars going to paid search) is ripening.

There are hundreds (thousands?) of dedicated search agencies and fully operational in-house teams with recognized delineation of roles. We’re not yet a fully mature industry, but there is a sense that we have plateaued — in a good way.

Advertisers have truly embraced this channel and understand the benefits and how it is priced. Early on, it was a battle for budgets, but I’m happy to say that I haven’t had to defend why paid search has shown up on digital campaign plans for years.

Depending on your experience level with paid search, your day-to-day responsibilities vary. Tasks that were challenging when you were first starting seem effortless now. But new duties are always popping up.

Below, I’d like to offer some advice to each strata of our industry. This may also help folks rather new to search better understand the career path ahead of them and the expectations that come as you take on more important roles.

Note: The years of experience listed are strictly arbitrary and just a loose guide for this post. I’ve met folks with two or three years’ experience who really know their stuff and have certainly run into so-called experts with many years in search who are still fairly clueless to the subtleties of the art.

Search Novice (first year)

Welcome to search! Right now, the most important thing for you to figure out is if you really even want to be in search engine marketing. It’s a growing field and should offer some great job security moving forward, but if you don’t get excited about optimizing accounts, analyzing spreadsheets and dropping CPCs like they’re hot, then this might not be the career for you.

However, if you are enjoying the little exposure you’ve had already to the medium, then it might be worth your time. I didn’t stay solely in search, but the lessons I learned while being 100% focused on SEM have helped me as a full-service digital marketer. Right now, just keep doing the tasks you’re given, keep asking good questions and get that AdWords certification.

Search Apprentice (1-2 years)

Now that you’ve been doing this for a while, it’s time to get deeper. If you haven’t been reading trade blogs such as SearchEngineLand.com daily, then you need to start.

Browse the headlines and read what interests you. Over time, you’ll start to figure out what’s going on and who the players are, and you’ll get a better feel for what this industry is all about.

On the work side, you should now be able to handle small accounts by yourself: pacing, reporting, optimizing, etc. Stewarding accounts shouldn’t be too hard for you even if you’re not always getting the high performance you’re shooting for. You should be building your own set of best practices that you can count on to work when you need them.

Search Marketing Analyst (3-4 years)

At this point, you’ve probably figured out that search, or at least the online advertising world, is for you. Congrats! There are a lot of people out there who haven’t found the right line of work, so you can consider yourself lucky.

Now that you know that you’re in this for the long haul, it’s time to really get serious. You should already be a leader on your team in some aspect, but you can now take the opportunity to step up. Mentor the newer folks and look to your boss for more responsibility.

You’re also going to be relied on to be in more important meetings with either clients or senior folks at your company. If you can’t shine and make your department look good, you’ll get left behind when it comes to promotion discussions.

We’re looking to you to really have mastered paid search by this point and start showing initiative to improve processes and build efficiencies without being prompted. When a new big account comes in the door, it should be a no-brainer for you to handle everything without needing supervision.

Search Marketing Manager (5-6 years)

Congratulations! You’ve obviously proven yourself very capable. Not only are you a bad-ass search analyst, but you also have the intangibles that organizations need to manage and supervise others.

However, just know that being great at search doesn’t always mean you’re great at managing. Hiring/firing, training, cost analysis, vendor review, etc., are all now hitting your desk. You’re going to be doing actual paid search less but rather ensuring your company’s paid search practice is as good as it can be.

You might even now be managing friends that used to be at the same level — it’s not easy going from bud to boss, but that’s the expectation. It’s important for you to make sure everyone under you continues to progress in their knowledge, even if it means forcing them to attend webinars and read the trades.

It’s your job to toe the company line and execute the vision of your director or VP while also being able to amicably offer your counter opinion when you feel something’s not going right.

Search Director/VP (7-10 years)

At this point, you’re probably not doing much day-to-day SEM. In fact, there’s a possibility that you don’t even know your AdWords login or the CPA on your best client. But that’s fine. Your focus should be on the department.

Your role is to help the team by making sure they’re working together like a well-oiled machine and that you are using your resources (people, budgets, time, etc.) in the best way possible. You’re also the main driver of your vendor partnerships, and if at an agency, providing a huge role in new business discussions.

Basically, you are search for your company. The things you think are important will now be what everyone under you thinks is important. They’ll be looking to you for guidance on the direction of the department for the next several years.

It’s absolutely crucial for you to have a deep understanding of not just search but the advertising industry (not just digital anymore.) Your company is counting on you to lead them to the promised land — don’t let them down!

Search Guru (10+ years)

Right now, there aren’t many SEM pros out there with 10+ years’ experience. But soon, there will be. Search really blew up around 2005, so by 2015 we’ll start seeing more and more folks with a decade or more of experience under their belt.

A guru might be a manager, a VP or even owner of their own company, but wherever their role on the team, success is determined by how much they make everyone around them better.

They’ll be patient with the novices and apprentices and help them avoid common errors. They should take analysts under their wing and show them the advanced techniques that only a decade of experience can offer.

It’s important for them to support their managers and provide great advice as these folks transition to from being employees to employers. For directors and VPs, the guru is a tremendously valuable resource for thought leadership and to help keep them informed on the search industry while their focus is being pulled in every direction.

Now that we’re ten years in, the roads have been paved. You should be able to form a fairly clear picture on the road ahead if you stay in this industry. Ten years from now, we’ll have college programs and folks will be getting degrees in either search or keyword marketing or some future combination.

Good luck, have fun and enjoy!

Ha! If it was only that easy!

Most of you reading this post have used one in the past. Have they worked well for you? Listen to any paid search veteran and you’ll hear stories of autobidders gone wild and delivering varying results.

I know that in the past, I’ve recommended in this column to just use auto optimization technology on your tail terms and manually optimize the important head terms of your account.

They’re good for managing low-impact keywords at scale but be careful not to just set and forget them. You may come back a week later and find that your pacing has slowed tremendously as the autobidder has paused every keyword except the handful that can meet your CPA goal.

Certainly, autobidders are not simple software. To build them right, you need to take into account dozens (if not hundreds) of variables and be able to slice and dice the data in order to decision upon bids inside paid search platforms.

To learn more about these tools, I spoke with Dr. Rob Cooley, Chief Technology Officer for OptiMine Software, a smart-cookie who has spent countless hours thinking about autobidders and trying to improve on them.

After speaking with him for more than an hour on this subject, I was wowed by the depth of what goes into cracking the autobidder puzzle.

Q: So, Rob, why do you love working on this complex issue?

Rob: My passion has always been creating data driven analytic applications and I tend to get bored if things are too easy. When I started working on my PhD in the mid 90’s there was this new thing called e-commerce that wasn’t very well understood from an analytics perspective. I dove in and found so many fun hard problems that needed to be solved in the space that it ended up being the subject of my thesis. Fifteen years later, I still haven’t run out of hard problems to work on.

Q: What has been your professional experience in this field?

Rob: I started consulting for e-commerce companies in the late 90’s while finishing my PhD, mainly working on things like shopping cart abandonment. In 2000 I joined a Xerox PARC spin-off that was focused on personalized search and acquired fairly quickly by Google in 2001.

Then, for the next eight years, I ran the technical operations for a data mining tool vendor. That gave me the opportunity to lead over 300 engagements where I had my hands on data to help solve a wide variety of marketing and advertising problems. The issue of pricing for online advertising kept popping up with customers, so in 2008, I decided to start OptiMine.

Q:  Before we begin, let’s define for the readers what an autobidder is.

Rob: My definition of an autobidder is a software application that automatically sets online advertising bids for every biddable entity (e.g. the “keyword” for paid search) in order to improve some performance metric.

Q: What are autobidders good for and what don’t they do well?

Rob: If done right, an autobidder improves performance while meeting critical business constraints, such as increasing profit while providing some minimum order volume, or increasing revenue while maintaining a minimum return on ad spend.

While they can provide some time savings, what they don’t do well is operate in a lights out environment without any human intervention. Someone with domain knowledge and an understanding of the business has to drive.

This is a common mis-perception about autobidders that I think comes from some early solutions in the market-place that were very black box in nature. In my opinion if you can’t steer the application, it wasn’t done right.

Q: Just how hard it is it to build a good autobidder? What are the variables or limitations that you have to consider?

Rob: It turns out that it’s incredibly hard to build a good autobidder. To do it right, you need to predict the future cost and value for each and every keyword. However, the vast majority of keywords have very little or no history of clicks or even impressions.

In addition, there is a lot of volatility even for the keywords that do get daily clicks and impressions. In technical terms, this is known as sparse noisy data.

Q: You mention there are different types of models out there, what are they and what are their pros/cons?

Rob: OK, here goes the deep dive…

To answer this, first there are some background terms you need to know.

Models versus Rules

A model based system uses past performance data to train statistical models to predict future performance. For example a model based system could predict the bids necessary to achieve a 200% ROAS. A rule-based system is typically a pre-defined set of reactions to certain situations. For example “if ROAS is less than 200% then lower bids by 10%”. In general model-based systems are predictive and rule-based systems are reactive.

Keyword versus Cluster models

 Within model-based solutions there are some that have different models for each keyword and some that group keywords together into clusters. The purpose of the clustering is to get around the sparse data problem by adding data from several keywords.

Global versus Local optimization

This is a really important distinction. Everyone throws around the term “optimize” or “optimal”, but there is a technical distinction between global and local optimization.

A local optimization simply bids each keyword or cluster separately from the other keywords or clusters. So if the goal is to maximize revenue with a minimum ROAS of 200%, every keyword is bid to obtain a ROAS of at least 200%.  A local solution won’t tradeoff low ROAS from one keyword with high ROAS from another.

A global optimization (referred to as a portfolio approach by some vendors) considers all of the keywords at once, assigning bids so on average the group as a whole maximizes a goal while meeting some constraints. It may turn out that one keyword can drive a ton of revenue at a ROAS of 180% and another at a ROAS of 220%, as long as the average ROAS is 200% the global solution will declare success.

You can spot a global solution by the looking at the available settings. If the words “maximize” or “minimize” are available as settings, then it’s a global solution. You generally can’t ask a local solution to “maximize revenue” or “minimize CPA” while meeting an additional set of constraints, you can only give it targets such as “provide a $15 CPA”.

Okay, now that you understand the ground work, I can answer your initial question.  The following are the four approaches on the market in terms of autobidding technology.

Global Keyword-Level

This is the gold standard or holy grail of bid management. The pros are performance and explainability. The cons are that it’s really hard to figure out since you need to predict behavior across a range of bids for each keyword and then solve a fairly nasty global constraint-based optimization.

Local Keyword-Level

In effect this is the approach advocated by Hal Varian. Bid each keyword separately based on the predicted value. The main pro is simplicity since you don’t need to predict behavior across a range of bids, just bid a percent of the predicted value. The main cons are limited settings and lower performance when there are constraints.

In a lot of cases, a local solution leaves money on the table. You can set a target but you can’t layer on multiple constraints. And while it may be achieving a ROAS of 200% you don’t know if it was actually possible to hit 250% that day because it is simply trying to hit the target, not maximize a metric.

Global Cluster-Level

Here, you still have a global optimization but models based on clusters of keywords are used to handle the sparse data problem. Some vendors are actually a hybrid of this plus the keyword-level global. e.g. they use keyword-level for head terms and clusters for the tail.

In either case, the pro to using clusters is model stability, meaning the results are repeatable. The cons are performance and lack of automation. The performance drop associated with clusters comes from the fact that each keyword is unique, and the value of extra data is outweighed by the loss of uniqueness. The other issue is clustering typically needs statisticians to manually tune the models, so cluster-based solutions are rarely pure software applications.

Rules-Based

This is probably the most common solution available. In theory, the pros are simplicity and understandability. I’ve found that’s often not the case since if you layer a set of 25 rules on top of each other it is very difficult to wrap your head around exactly what will happen to the bids. The main con is performance. Because of the reactive nature of rules, they can be very good at what I call profit protection, but they rarely lead to optimal results.

Q: There are seemingly thousands of these technologies out there, whether inside licensable tools or in some home-grown, proprietary platforms.  Where do you think many of them stand in terms of effectiveness and why?

Rob: I think my opinion of the effectiveness of the various solutions out there is best summed up by the fact that I chose to quit my job and found a company at the height of the financial crisis because I was confident I could create a better solution.

In terms of why, I have a strong opinion that you need a solid academic foundation in data mining & optimization techniques, strong domain experience with online advertising, and a lot of field experience with actual data mining and analytic applications to put together a viable solution. From what I can see, a lot of creators of bidding solutions are missing one or more of those three key points.

Q: So what’s the autobidder at OptiMine?   Why do you think you have something special over there? Does it have a cool name like “Conan the Keyword Destroyer” or “Bidder Bidder Chicken Dinner”?

Rob: Sorry, no cool name. It’s just OptiMine Bid Management. It uses a Global Keyword-Level approach per my answer above. The reason we think we have something special is that we regularly improve performance (profit, revenue, ROAS, etc) by 25% or more in controlled tests against other technologies. We haven’t lost a competitive test.

Q:  Can you share some results on how you’ve compared to competing platforms?

Rob: Here’s an example for each of the three competing types described above.

OptiMine vs Global Cluster-Level

The goal was to drive as many new accounts a possible at a fixed CPA. Against a competing global cluster-level solution, OptiMine drove 216% more new accounts at the same CPA. The key issue here was the age of the clusters. In this case they hadn’t refreshed their clusters and the keyword grouping was simply obsolete. There were a handful of good keywords hidden in clusters of bad keywords. The act of separating those out and bidding them up was what led to the volume increase.

OptiMine vs Local Keyword-Level

The goal was to drive as much profit as possible while maintaining a minimum amount of revenue. Against a competing local keyword-level solution OptiMine drove 37% more profit. The key issue here was seasonality. The local solution had much simpler models that just didn’t pick up a declining seasonality as fast as OptiMine.

OptiMine vs Rules-Based

The goal was to drive as much revenue as possible at a fixed cost of sale (inverse of ROAS). Against a rules-based approach OptiMine drove 30% more revenue at the same cost of sale. The key issue here was tail terms. The rules put in place were simply too conservative for the tail terms. OptiMine drove 159% more revenue out of the tail which led to the overall 30% increase in revenue.

Q: So, I saw the announcement that Adobe is using OptiMine’s technology in SearchCenter?  Sounds exciting…can you talk more about this relationship and how it will work?

Rob: Yes, the next release of SearchCenter will allow their customers to use the OptiMine bid management technology. Essentially, it’s Adobe’s version of our interface. The back-end number crunching is still done by OptiMine, but instead of the OptiMine UI the features and functions will be seamlessly integrated into the SearchCenter UI.

I balked. But I took the bait. “What’s that even mean, dude?” I said.

Josh went on to explain that all signs point to the fact that display is going to become the dominant advertising channel online and that search retargeting will become the dominant tactic for display advertising.

Thus, display is changing the value of search as search will become even more valuable for targeting than it has been for the last decade with just buying text ads on search engines.

“Soon, not doing search retargeting will be a fireable offense for an advertising agency,” Josh boldly explained, and continued his argument:

“The advertiser will ask: ‘You mean a user went on a search engine, raised their hand saying they want a new cell phone, and the user only received one text ad via the search engine, next to our competitors text ads, and on a page that the user likely only spent 30 seconds? You never followed up with that user beyond the search engine, in a one-on-one setting on their favorite publisher, and with a picture of our product?  You’re fired!“

Okay. Got it. But being the skeptic I am, I decided to dive into the evidence to see if he was right.

Exhibit A:  Display Will Outpace Search

Don’t look so surprised. We always knew this was inevitable. The fact is that paid search is a pull medium as it requires users to search on an engine to trigger an ad. Display is a push medium which simply requires page views to deliver [normally multiple] ads to the user.

Bottom line:  page views online are growing exponentially faster than searches on search engines.

Check out the eMarketer chart below. Search spending is on pace to almost double in the US by 2016 whereas display almost triples. If this chart were to extend out to 2020, there’s no doubt (based on current market conditions) that advertisers will be spending more on display by that time.

Exhibit B:  Display Offers More Impressions Than Search

“Search retargeting gives advertisers a second chance to reach consumers who might have not been ready to transact during their search experience,” says Harrison Magun, SVP of Paid Media & Analytics at Covario. “Search retargeting is the gateway to display opportunities for search advertisers.”

Let’s do some fuzzy math. On a Search Engine Results Page (SERP), we’ll assume that on average, there are six paid listings on a page. One million unique user searches for Keyword X would equate to (1 million searches x 6 ads per SERP) six million ad impressions, right?

However, let’s say that there are twenty advertisers who would want to target users who searched for Keyword X.

If they all use search retargeting and were able to reach those million users an average of five times, that would be (1 million searches X 20 advertisers X 5 Frequency) one hundred million ad impressions.

So, in this example for the same one million searches, traditional paid search would deliver six million impressions versus search retargeting with one hundred million impressions.

In the case of highly competitive keywords, there may be literally dozens and dozens of advertisers using search retargeting to push well over ten to twenty impressions to those users.

In that case, this could mean hundreds of millions of impressions chasing those one million users who searched on Keyword X.

Exhibit C:  Search Retargeting Works For The Same Reason Search Works Well

One of the main reasons why paid search works so well is the expressed intent that’s passed from the user to the advertiser. We can reasonably assume that someone is interested in grilling when they search for “buy bbq grill” or are in the market for a Hawaiian vacation when they search for “best deals on cruise to Hawaii”.

The keyword is a huge indicator of what kind of messaging to which a user would respond best. Right now, with many display ads being loosely targeted to user site behavior or geographical location, search retargeting’s ability to target to users who have identified their intentions via searched keywords is proving to work.

According to Michael Davis, Senior Media Planner at Razorfish:  “By using site and search retargeting with Magnetic, we’ve seen glowing results for our clients. This targeting combination allows us to leverage both customer acquisition and customer retention strategies online.”

Exhibit D:  Search Retargeting Is Growing

This tactic is blossoming as we speak. Besides Magnetic, other search retargeting companies already in this space include Chango and Simpli.fi. As well, Criteo is getting involved and so is mediaFORGE.  The big player that seems to be making moves on this front is Google.

As reported in AdAge recently, Google is “quietly building an exchange for buying and selling data.”  There’s no definitive reports that this data exchange (dubbed “DDP”) will include search data, but the Big G must know that they’re sitting on a virtual fountain of search retargeting gold as the leading search engine in the world.

Judge’s Decision:  Display Is Changing The Value Of Search

Based upon the evidence, I’m going to have to agree with Josh Shatkin-Margolis on this one.

• It’s inevitable that display’s budgetary importance will outgrow paid search over time
• Search retargeted display will offer many more times the impression opportunities per keyword search than traditional paid search advertising
• Due to the passing of the user intent [keyword] to the advertiser, search retargeting could very likely become the dominant targeting tactic in the display ad space
• The search retargeting industry seems to be growing, not slowing

Josh is not saying that search will become irrelevant. If anything, search will always be a very important part of digital advertising. It’s still growing, albeit slower than display overall, but it’s definitely here to stay.

However, what he’s saying is that because of the reasons listed above, marketers will begin to see the value of search to be both on the SERP and off the SERP. As display grows (and search retargeting grows with it) search extends far beyond where it is now.

Thus, display is changing the value of search which should be good news to the search pros out there! It’s going to make search even more important than ever.

It’s almost like trying to beat the stock market – only someone with a crystal ball could really nail it on the head every time.  However, with some historical information and a few calculations you can get fairly close to figuring out the right bid to use for each keyword.

Today, let’s check out how Google Chief Economist, Hal Varian, teaches bidding in his Google AdWords Bidding Tutorial. He explains that proper bidding is anchored in a simple cost vs. profit analysis.

All you have to do is make sure you bid in a way so that you bring in more revenue than you spend. It’s easier said than done and you will certainly have trouble bidding correctly for every keyword, but as long as you follow this standard process, you should be able to bid well more often than you bid poorly and come out ahead in the end.

Basic Bidding

The first step is to figure out the conversion rate of the keyword. This should be a fairly easy metric for most SEM pros to figure out, but for any newbies to this industry, your conversion rate is your Total Number of Conversions divided by your Total Number of Clicks. So, if a keyword drove five conversions from every one hundred clicks, your conversion rate is 5% (5 conversions/100 clicks).

Once you know that keyword’s historical conversion rate, you then need to figure out your Value per Click (VPC).  The VPC calculation is your Max Profitable Cost Per Action (CPA) multiplied by your conversion rate.

What is your Max Profitable CPA? It’s the most you’re willing to pay for that action/conversion and not lose money. In his tutorial, Varian recommends not padding that number for profit but rather setting it just at the break even point.

For example, if you’re reselling leads for $100 each, you may set your Max Profit CPA at $100. You’re in this to make money, right?So, you may feel the urge to bring it down to $75 to make sure you make $25 per lead, but try to resist that temptation.

Think about this: what if I told you I could get you a million leads for $90? Would you really turn away $10 profit per lead? If you used $75 in your calculation, then you may be end up optimizing out of that profit. For this exercise, we’ll go ahead just use $100 for the Max Profitable CPA.

With a Max Profitable CPA of $100 and a conversion rate of 5%, your Value per Click is $5.00 ($100 x 5%). Basically, at the end of the day, each click of that keyword is going to be worth five dollars to you. This also means that if you paid five dollars per click, you would just break even. To make a profit, you have to pay less than your Value Per Click.

Varian makes a simple observation that your actual Average Cost Per Click (Avg. CPC) in almost all situations is going to be lower than your Max Bid. So, using that rule of thumb, Varian concludes that bidding at your Value per Click should always lead you to a profit. However, you may find yourself bidding over your Value per Click often because your Avg. CPC still is below that threshold.

The five dollar bid now puts you in the ballpark, but I would argue that doesn’t mean you’re in the game. Since raising your bid usually increases your Click Thru Rate (and conversely, clicks), does that mean by raising your bids will automatically increase conversions?  Of course not. And will that incremental cost lead to more profits?  Not always.

In fact, in the video (see screenshot below), there’s an example where lowering the five dollar bid to four dollars actually creates more profit because, even though it generates less total revenue, the higher costs associated with a higher bid eats into the net revenue.

Advanced Bidding Using Incremental Cost Per Click (ICC)

To truly win at paid search, you need to take that starting bid and optimize to an optimal bid in order to maximize your efforts.  But how do you do that?

To accomplish this, the tutorial presents the metric, Incremental Cost per Click (ICC), which is the Cost of Incremental Clicks divided by the Number of Incremental Clicks.

In the example below, you can see that increasing your $4.50 bid to $5.00 does get you eighteen more clicks, but at an incremental cost of $103.15. When you divide them, that’s an ICC of $5.73.

And once you calculate out your ICC for other bid levels, you can easily see which bids come closest to your Max Profitable CPA without going over. In this example, $4.00 is your optimal bid.

To calculate out your ICCs, you would need to have the performance data for your keyword at various bid levels. Back in the olden days of search (circa 2004), we would have to build tests to set different bids and then analyze the data. But now, we have Google’s Bid Simulator that can provide some of this data.

What is the Bid Simulator? From the AdWords Help article on this topic:

The Bid Simulator enables you to see the advertising results you could get if you used a different maximum CPC bid for your keyword or ad group. It doesn’t predict the future, but it estimates the click, cost, and impression data that your ads would have received over the last seven days had you set different bids.

For example, suppose you have a maximum CPC bid of US$1.00 for your keyword, and you wonder what results you’d get with a bid of US $1.50 or US $0.75. The bid simulator shows the clicks, cost, and impressions you would have seen with these other bids over the last seven days.

To generate data, our system analyzes the auctions on Google and the Search Network from the last seven days. The bid simulator uses internal auction data, such as Quality Score information for you and your competitors, to estimate where your ad would appear and how frequently it would have gotten a click. The bid simulator keeps everything the same except for your bids. For example, it uses the same values for your competitors’ bids, traffic on Google, and Quality Score.

Thanks AdWords! To access the Bid Simulator in the UI, go to the Keyword tab and look in the Max. CPC column. Click the little graph icon and you will see a table like the one below that shows you clicks and impression based projected out at various bid levels. You can use this information to calculate your ICC and find your profit-maximizing optimal bid.

Here’s an example of what you’ll see when you click the Bid Simulator icon in AdWords:

Looks good on paper.

Easy, right? Ha. Well, on a training classroom’s whiteboard, it works out just fine. But we all know that there are many real-world variables that will disrupt this perfect plan of action.

Remember, the Bid Simulator is just a simulator. There’s no guarantee that your account will act in accordance to its projection. As well, even Varian admits that you should check the tool on a regular basis as the numbers might change due to changes in the competitive landscape of your keywords.

There’s also outside factors such as seasonality, economic downturns, new innovations in the marketplace, etc. that can drastically affect your conversion rates and Max Profitable CPA.

This is where the real value of an SEM pro comes in to be able to take the very quantifiable approach that Varian outlines and weigh it with the unquantifiable instincts that expereince can add.

Added value awesomeness from TenScores:  The Bid Optimizer tool.

In this column earlier this year, I interviewed TenScores founder Chris Thunder regarding his quality score tracking tool. As you know, Adwords doesn’t currently provide historical data on Quality Scores.

TenScore’s tool allows you to link your AdWords account and then pulls down your QS daily so that you can track the progress of your optimization efforts. This is a really light, cost-efficient tool that could make a wonderful addition to the SEM toolbox for anyone serious about improving their Quality Scores.

Recently, I came across the TenScores free Bid Optimizer tool, which is a very simple online calculator that lets you insert your keyword’s Bid Simulator data and then graphs out your various ICCs in order to figure out an optimal bid.

It’s a really great tool that takes a lot of the math out of the bid calculations and makes it actionable. Just pop in your data and you’re ready to roll.

Bravo to TenScores and Chris!  Here’s a screenshot:

Good luck to everyone out there on your bids!

I started buying keywords on Overture with a company I wasn’t even doing marketing for, which got me into some early e-commerce activity, and then to the agency world.

It definitely feels like a marriage…there have been some good times, some bad times. Some amazing times, and some really down in the dumps times. I’m nowhere near the person I was when I started and neither is the search biz.

We’ve both grown tremendously in the last decade and I don’t think either of us realized just how big this thing was really going to be. I feel like a proud spouse watching my significant other’s success with a huge smile.

I’ve learned a lot in ten years too — usually by making mistakes. Aptly, here are ten insights I’ve gained in ten years. Maybe some of you folks new to search won’t have to learn like I did — the hard way.

1.  Test. Optimize. Repeat.

This was one of the first major best practices I learned all of those years ago. Everything you do in paid search is just an experiment waiting for its testing period and analysis. No tactic is set in stone. No result is considered the best; performance can always get better if you can get smarter.

2.  The Most Granular Level Is Not The Keyword

I’ll credit SearchRev with really hammering this home when I worked with them in 2008. Their paid search management tool was the first one I had seen that allowed each ad or keyword to have multiple landing pages loaded in.

Just the same way you would test multiple ads with each keyword, why not test multiple landing pages for each keyword/ad combination –  because conversion rates will definitely be affected and optimized at that level?

Now, years later, the most granular level of paid search might be more like:  keyword + match type + bid + geotargeting + ad + landing page.

I feel that if you treat the keyword as the most granular level, you’ll make clunky, high-level adjustments to the campaign elements which could end sending ripple negative affects across the account.

3.  Put More Into The Upfront Work

Some of the grunt work of SEM such as research, keyword building, ad group and campaign grouping, etc. have a tremendous effect on the success and management of the account. If you put really great effort into the research, the insights you uncover will pay dividends during the flight of the campaign.

Sometimes, we overdo the research just to ensure we’ve done enough. And just really thinking through the architecture of how you build your ad groups and campaigns before you begin can make future reporting, analyzing, and optimization more efficient.

4.  Short Head/Fat Middle/Long Tail

I detailed my philosophy on this in a column a few years ago and it’s still the way that I approach paid search management.

The Short Head are your top handful of terms that really drive most of your spend and conversions — usually these are your branded and product terms. I recommend looking at those every day manually as those have the biggest effect on your account.

The Long Tail (at least how it’s been bastardized for SEM from Chris Anderson’s original vision) are all of those hundreds or thousands (maybe even hundreds of thousands) of keywords that just get minimal searches per month. Usually, some sort of tech solution or management tool is best utilized to handle these at scale using auto-optimization technology.

The Fat Middle terms are the ones between the Short Head and the Long Tail.  They don’t necessarily warrant to be checked every day,but they’re certainly more important to the account the tail terms and do require some manual attention on a reoccurring basis.

5.  Don’t Get Complacent When Performance Is Great

Even the most professional SEM pros probably struggle with this one. You spend months, if not years, getting the account in a great place and its performance is knocking the socks off your boss (or your client). You just don’t think you can squeeze any more water out of that stone, especially with other accounts waiting on optimization which may not be performing so well.

To combat this at my agency, we institute an optimization plan of attach at either the campaign start or on a reoccurring basis for ongoing accounts. The optimization team can go off the menu at any time if they feel that an account doesn’t need the scheduled optimization that week, but they have to replace it with something better.

6.  Never Stop With Your Education

Another best practice that I learned fairly early… Things change so often in this industry that you have to stay on top of it, even if you think there’s nothing new going on.

I especially credit Google and their aggressive product roadmap with AdWords. Their innovation schedule is unbelievable and that product grows in power every year it seems. Plus, as the digital marketing industry in general grows and evolves, it affects the way search is used and perceived.

7.  Have The Right Tech Stack

Get good at evaluating SEM vendors and understand what’s on the market and available to you. Yes, you could do paid search with just AdWords and AdWords editor. But I think there are major limitations if you don’t use some of the advanced tools that are out there. Some of them are even free! Check out my Giant List of Keyword Tools post from last year for some ideas.

8.  Get Up To Speed On Your New Tools As Quickly As Possible

This took me years to learn. Many times, when you start with a new tool, there’s so much to learn and you’re already busy with other things, that the tendency is to phase out the onboarding process.

However, I’ve completely turned around on that way of thinking. What I learned was many live, account mistakes occurred during the onboarding process that could have been avoided with training. Also, time is money. You’re probably bringing in this new tool for a reason—so whatever that reason is, why not get it going ASAP?

9.  Everyone Needs To Be Great (Not Just Good) At Excel

For the lucky few of you out there that have been able to completely wean themselves off of Excel, I applaud you. But most of us have to spend some at least some time in spreadsheets.

That being the case, everyone in your organization (except maybe the cleaning lady) needs to be great at Excel. We utilize on-demand Excel video training as well as our own on the job learning.

10.  The Search Industry Is (Still) A Small World

Today’s colleagues may be tomorrow’s clients. Your local engine rep could be at another engine next year. This is the nature of our little world and it’s a good idea to read the trades, follow the industry (hires/fires/acquisitions/etc.) and attend conferences.

Most importantly: be nice to Google.

I’ve had my fair share of issues and screaming matches over the years with them, but the hard fact is that Google is search at least for now and the foreseeable future. It’s not always easy to work with them, and I’m not always easy to work with either, but if I want to work in search marketing, I have to have a good relationship with the Big G. I’m glad to say my relationship with them is better now than ever.

Ultimately, it’s still just a company of people and they do hire great folks. Usually the issues are more about policy the individual folks have zero control over, so they’re just as frustrated as you when they can’t help you out.

Stock image from Shutterstock, used under license.