Raven Tools To Remove Scraped Data To Maintain Access to AdWords API

google-adwords-square-logoWhile there has been lots of talk about a Google crackdown on developer access to the AdWords API, there have been few details about what the disputed issues may be — and why they’re coming up now, with tools-makers like Raven and SEOmoz that have had access to the API for years.

Now, Raven Tools, which was warned about possibly having its API token revoked back in November, is opening up about what’s going on in its case, at least. The issue: Raven has long been using data scraped from search engine results pages (SERPs) to display ranking and keyword data for its customers.

While the company has been doing this for years, and says it’s successfully gone through two previous Google audits, the search engine giant is now choosing to enforce the anti-scraping provision in its AdWords API terms of service. Google has previously denied that it’s cracking down, or doing anything new other than continuing to enforce its terms of service.

Given Google’s new hard line, Raven has chosen to stop displaying the scraped data — SERPs rankings and keywords — within its tool set, in favor of keeping on good terms with Google, and retaining access to the AdWords API. Based on comments to its blog post, the reaction has been mixed, with some saying they understand the desire to stay on Google’s good side, while the majority of commenters complained that the scraped data is much more valuable than the AdWords API data.

While one of the appeals of Raven for marketers has been getting a lot of different types of data in one place, it seems Google’s stance — to remove AdWords API access from developers that are also scraping data — will likely prevent an all-in-one solution from existing.

Raven will be removing the SERP Tracker data as of January 3, and will allow customers to export their data until January 2.

We’ll have more on this issue coming up later today, so stay tuned. 

 

Related Topics: Channel: SEM | Google: AdWords | Google: APIs | Top News

Sponsored


About The Author: is Executive Features Editor at Search Engine Land and Marketing Land. She’s a well-respected authority on digital marketing, having reported and written on the subject since 1998.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Henley Wing

    In my opinion, this greatly reduces the value that Raven Tool provides. After all, you can’t improve what you can’t measure. And if you can’t measure rankings, how else are you supposed to know you’re making the right changes?

  • http://searchengineland.com/ Danny Sullivan

    Rankings don’t tell you anything other than you rank for a particular term, not that you’re getting traffic for a wide-range of terms you might not be tracking. And given how rankings vary so dramatically by person and location, they mean even less.

    If you want to know how the things you are doing are working, you look at the traffic you’re getting. That’s your success metric.

  • Travis Prebble

    It represents an insta-cancel for me, especially as they’re also removing SEMrush from the keyword research tool. I can use AdWords to manage and track AdWords performance. What I didn’t have was a rank tracker, ergo the subscription to Raven.

    Regarding measurement, it’s still possible to measure traffic and resulting conversions from search engine visits, so one can get a decent idea whether or not their efforts are worthwhile, but the results are a bit mushy. Keyword rank was such an easy thing to communicate: “Last week, we were 50. Now we’re 5.” But even that concrete number is losing relevance since it is difficult to weed out all the results massaging Google does these days to improve relevance for the searchers themselves in addition to simple keywords.

  • http://twitter.com/RonSansone Ron Sansone

    Raven provides great customized reporting. I like how it displays Analytics data, and the rankings display in the reports were nice and clean. But yes, without SERP Tracker there’s no point.

    There’s a ton of great tools for combing through and displaying Analytics, Link, Keyword and AdWords data already. The reason I liked Raven in the first place was because it integrated the data sets into a digestible, unified report. The weekly rankings scrape was both a time-saver and a way to ensure regularity of tracking. Without that element in there, I’m forced to use an outside tool for rankings data, which ruins the entire point of unified performance reporting.

    Bottom line: I can make pretty reports without Raven Tools. Raven Tools made reporting successes much easier because of the passive rankings monitoring of SERP Tracker being layered on top of the other data sets.

  • Henley Wing

    Traffic and rankings may be the same for publishers that rely on display advertising such as AskMen, or iVillage, but for most businesses, there’s a huge difference between getting 1000 visitors from a highly relevant, converting keyword, and getting 1000 random visitors that came from Twitter.

    Rankings also are an easy way to let you know whether a SEO campaign/agency is working effectively. If your ranking went from #5 to #1, you can attribute that to SEO, and add more to their budget. If your traffic went up, it could’ve been attributed to PR, or marketing, or advertising.

  • http://twitter.com/RonSansone Ron Sansone

    Yes, this. We can all say “rankings are dead” until the cows come home, but that doesn’t change the fact that they’re a bellwether for aggregate performance despite personalization. No SEO worth their keyboard is tracking rankings and traffic as an either/or proposition. It’s a matter of examining ALL signals in concert. And we know that keywords in a stronger average position over the aggregated user Search experience will result in more qualified traffic organically driven.

    As such, rankings tell you a lot when framed against other metrics. This really does negate much of the value of the tool set unfortunately.

  • http://www.facebook.com/conortreacy Conor Treacy

    It looks like I’m headed back to using Excel Spreadsheets and multiple programs to gather the data for clients. Not having SERP results in the report and then also not having SEMRush pretty much cuts out 80% of what we use with Raven at this point. it’s back to scripts, exports and imports into Excel for me it looks like :(

  • Paul Dillinger

    Makes no difference to me. I never liked Raven’s SERP reporting (though it seems I’m one of the few). It’s more work, but I can get better data using other tools. I do lots of local optimization and work with local listings all over the country. I find rankings for sites vary greatly based on region and local competition for most of terms I deal with. If I only used Raven’s ranking data I would be using about 10% of what I currently get. I don’t spend much time worrying about SERPs, but looked at over time they do help in determining overall trends and troubleshooting.

  • http://www.seo-theory.com/ Michael Martinez

    There is no place for tracking “rankings” in search engine optimization and has not been for years. The Raven blog pointed out the flaws in tracking rankings. You’re making business decisions on the basis of amalgamated data that does not reflect reality.

  • Justin

    This article says that without scraping, Google is preventing an all encompassing data source from existing. That is not accurate. BringShare has all these data sources including Adwords and SERPs but does it via direct integration with all the search engine API’s. Yes, that means the search engines get paid for the data BringShare pulls but it also means there is no scraping. Full disclosure: I’m the Cofounder & CEO at BringShare.com

  • http://twitter.com/weboptimist weboptimist

    True, Danny, but for many of us, the CEO still wants a ranking report, no matter how many times we argue your point.

  • http://www.geekpoweredstudios.com Guillermo Ortiz

    Without rankings Raven will definitely fall short as the go to all-in-one solution, but if you plan on dropping it solely for the lack of rankings then you were probably following the wrong metrics to begin with. Rankings are just one of many metrics that easily and visually show campaign progress to a client. The problem is that as an industry rankings are talked up and made to be holy grails when in reality they often just cater to the egos of clients. Ranking #1 for a high volume, low converting, crap keyword should not be the end goal – growing/increasing business should be. That being said, let’s be real, as we know we all have certain clients that only understand value through rankings so the metric won’t be going away anytime soon. Kudos to the Raven team for making one hell of a tough choice.

  • Henley Wing

    @Michael_Martinez:disqus Tell that to the clients who want the data. That’s like being a financial data provider saying to portfolio managers: “you’re just gambling, our analytics/financial data won’t help you anymore than throwing darts in the wall!”.

  • http://www.facebook.com/profile.php?id=588188254 Jim Hodson

    I have to agree with Danny. Rankings never were a good KPI/metric. But they have been rendered 100% worthless over the last few years by all of the personalization and localization that Google does.
    I just visited a client for a meeting yesterday. I reviewed the ranking report their web design company gives them every month. A quick glance at the ranking report and check of AdWords’ keyword tool revealed that not ONE of the top 10 non-brand keywords that they were checking rankings for (and boasting about position #1, #2, #3 rankings) had any estimated exact match US searches in a month… none!
    Other than monitoring a few “money” keywords to detect huge swings in rankings that might signal a filter or penalty or algorithm change of some sort, rankings have been pretty much worthless for several years now.
    Retrain your CEO. Explain to him that searchers for the same term all over the US or world for that matter yield different results. Explain that it is a meaningless report. Just because you rank well for a keyword phrases doesn’t mean there is significant search volume, click throughs, or conversions.
    Then show him/her a killer analytics report similar to ranking reports but that track actual visitors by top XXX keyword over time like you did your rankings. Amend that with additional reports tracking visits resulting in conversions by the top XXX keywords.
    If your CEO can’t see that the analytics report is 1000x more valuable than any ranking report he/she used to get then I’d petition your board to appoint a new CEO that knows how to run a business. LOL

  • http://twitter.com/RonSansone Ron Sansone

    You can absolutely say rankings are a poor KPI. Honestly, they’re supplemental and used to examine correlations with analytics data and tactics most of the time. And no one is acting as if rankings are ONLY thing they pay attention to — far from it.

    But let’s break this down to the practical stuff: Why would people keep paying for a service that’s providing less information? The only major difference between Raven’s Pro and Agency Pricing plans were the number of keywords tracked. Even if people keep with it, I’d expect to see a lot of downgrading.

  • http://twitter.com/RankWatch RankWatch

    Paul, you might want to try RankWatch as its the only tool which tracks rankings in google locals.

  • http://twitter.com/TylerJKelley Tyler Kelley

    You should put your Agency pricing on your site. Raven’s $99 price point for small agencies is a big selling point. Your site says contact us for pricing which makes me think it’s going to be a lot more.

  • Paul Dillinger

    Looking in to it now. I’ll let everyone know what I find. Thanks.

  • Ulysis Cababan

    I agree, I think I will start preparing some excel sheet for the clients as early as today. Because after new year we will have a big bang of work to do.

  • Paul Dillinger

    After trying it out, it’s not exactly what I need. It seems to be way off on rankings. It looks almost like it’s using Google AU even though I specified US. A few national 1st page terms (verified by my own tools and from Raven) are not showing in top 100. Also the local stuff isn’t very intuitive. I’ll stick to my current tools. Good idea though, keep working on it.

  • http://www.SEOwhiz.com/ Bob Costello

    Okay, now I’ll promote ‘site traffic’ as the relevant metric. “You Got Traffic!”
    And, yes, I’ll be cancelling the subscription to Raven.

  • http://www.facebook.com/profile.php?id=588188254 Jim Hodson

    I agree. This is going to hurt Raven and others like them.
    Personally, I’ve never been a big fan of SEO tool “suites” anyway. You typically get a bunch of mediocre tools with lots of limitations when you do this.
    For example the link data available through Raven is pulled from Majestic SEO. Why not just subscribe to Majestic? My one exception is SEOmoz. And I honestly think most of their tools are mediocre as well. I only subscribe to SEOmoz so that I can have access to Open Site Explorer to augment what I get from Majestic SEO and a few other sources.
    Lots of specialized tools are always better than 1 suite trying to be all things… and limiting the number of domains you can use it for, the number of keywords, etc. Suites = Jack of all trades… masters of none.
    But there is a cost related to paying for several specialized tools instead of one suite that many not serious about SEO are simply not willing to invest.

  • http://www.bringshare.com/ Justin

    Good point. Our minimum is based on 10 dashboards. Adwords, social, email and the master dashboards are $4.90/piece. Seo dashboards are $10/piece per month. So starting is $49-$100 depending upon what dashboards you need.

  • Nick Cuttonaro

    @Michael_Martinez:disqus Seriously? I completely disagree with you. Tracking SERP movements and rankings is a big part of understanding the cause and effect of actions you’re taking within an organic SEO campaign. Saying there’s no place for tracking “rankings” is a hard pill for swallow for an SEO working on a reputation management campaign or a small business ranking #1 on Google for their target keyword.

  • Bobby Honess

    Seems like the FTC should take a look at this. Google basically used their monopolistic position to strong arm Raven to remove this data. While they can crawl the web, Raven cannot crawl Google – and now their business is likely to take a huge revenue loss as a result. Google should make it easier to get this information if they have a problem with scraping. This is a huge issue and another example of Google acting like a monopoly. Less money on SEO, more money on adwords.

  • Andy Atkins

    I’ve just taken a look at your site, Justin. I have 2 questions. One, aren’t you going to get caught up in Google’s TOS as well (my understanding is that no-one has direct access to Google SERPs, even via GWT average rank)? Two, how much will it cost per client to broadly replicate the facilities in Raven Tools?

  • http://www.bringshare.com/ Justin

    Thanks, Andy. I don’t want to get too specific here for competitive reasons but we do have direct access and we pay Google every single time we check the rankings for a keyword. The same goes for Bing/Yahoo. The SERPs data is actually our biggest cost in providing data to our users and is why we have to limit the number of keywords you get to 25 for the $10/mo price. To get the ranking, we send a query to Google via API and request that they return the ranking result. Regarding replicating Raven, while we have a lot in common, philosophically we’re quite different. We do not intend to ever be a “doing” tool where you can manage campaigns, research keywords, etc… Our mission is to be the very best “viewing” tool that is a simple compliment to any agency or consultant saving them time and money through great client reporting automation. We have LOTS of new features rolling out in the coming months that are all focused on making our reporting/dashboard tools even better.

  • http://twitter.com/TylerJKelley Tyler Kelley

    So that’s $49-$100 per URL tracked? Per client? I like what I see but I am a bit confused. Let’s say I have 25 clients and I track adwords, social, analytics, and rankings (100 keywords each). What’s that going to cost me? For comparison purposes, this is $99/month with Raven.

  • http://www.seo-theory.com/ Michael Martinez

    Which rankings are you tracking? The ones in New York or the ones in Los Angeles? Are you tracking your mobile phone rankings or desktop rankings? Are these the rankings from a west coast data center or an east coast data center? Are they the logged in rankings or the logged out rankings? Are they the rankings from 2AM in the morning or from 12PM at midday?

    Rankings are meaningless.

  • http://www.seo-theory.com/ Michael Martinez

    “Tell that to the clients who want the data. ”

    Actually, I DO.

  • sergiuliano

    You should also try KeySR.com for free, as it is the most powerful local rank checker available on the internet at this moment

  • Andy Atkins

    Thanks for the comprehensive reply, Justin. It’s an interesting service, but is going to be difficult to cost-justify when you’re an agency with multiple SMB clients, some of whom (with 2 or 3 domains) need 50-150 keyphrases tracked across 2 engines. I will keep an eye on how you progress.

  • Matt McGee

    Okay, folks, let’s put this part of the discussion to rest. The comments section on our article isn’t a place to conduct business, so please take it somewhere else. Thanks.

  • Dorothea Kettler

    Google may have taken down Raven, but there are still many other tools that get their data through scraping and that are probably not going to comply :) For now I’m going to stick with the SERP tracker that was recently released by KeySR.

  • Asif Billah

    Great points.Title tag is really important and can make rank to the top page even many other techniques of SEO are wrong for less competitive words.
    Your theory is good

  • Henley Wing

    The average or aggregate rankings, Michael. Let’s not over-complicate things. Let’s say you own Zappos. Zappos ranks #1 for “dress shoes” and “casual shoes” and “brown loafers”. Ok.. they might be #2 in LA, or #4 when you’re logged out. But overall, they rank in the top 1-3 for most shoe terms..

    Now suppose I’m Google, what if I decide to drop you 100 spots, so you rank in the 10th page instead of the 1st? How about we do that, Michael? No? But I thought you said rankings were meaningless!

  • Henley Wing

    I agree. Rankings, especially for targeted keywords are more important metric than overall traffic. I mean, haven’t we been through this over and over? Are “eyeballs” what ultimately matter now?.. again?

  • Joshua Mccoy

    All I can say is: shame on you Google.

  • http://twitter.com/NowClicks Ryan Patrick

    Seems like raven doesn’t have much of a choice…

  • http://twitter.com/NowClicks Ryan Patrick

    I guess it’s difficult doing that now with Google Analytics as 40% of search queries are encrypted. Webmasters is still beneficial but Google is making things more difficult for the SEOs…

  • http://www.seo-theory.com/ Michael Martinez

    Aggregate rankings tell you nothing about search performance. To present such a naive analysis to a client is a disservice. There is no “overall” ranking for anything.

  • http://www.adela.vn/dich-vu/thiet-ke-logo.html Thiet ke logo

    I also hope the issue concerns all help. Thank you very much

  • Keesjan Deelstra

    @Danny: Thats why we advocate ‘Traffic Share’: a new KPI that represents the estimated percentage of the maximum amount of visitors for each keyword. A real gem when it comes to calculating a quick business case. See http://www.seoeffect.com/seo-tools/monitor-seo-trends.php

    But on tremendous pressure from our clients ;-) we now also have SERPS as KPI for those clients with bosses in need for rank reports…

  • Unbound Marketing

    I’m not surprised Raven point out the flaws in rank tracking, just as they’re losing this functionality in their tools!

    There is a place for rank tracking i think, but only a small one in the corner of a big SEO room.

  • Johannes Selbach

    Here’s a comparison of Raven,SEOmoz and SEOprofiler:

    http://www.seoprofiler.com/seo-tool-comparison

    Disclosure: I’m the co-founder of SEOprofiler.

  • Holly Ares Snyder

    Nail on head, @dannysullivan:disqus . I’m trying to make the case that we’re going to have to spend a bunch of time dealing with this issue either way; better to spend it re-educating clients (overdue) and polishing up our other metrics than to spend it finding and testing alternative ranking solutions.

  • LaurieOToole

    One of the biggest challenges I see is that SEO agencies’
    clients still ask for rankings. In many instances the customer is too busy to try and understand the intricacies of some other metrics that we’d like them to consider; and in some instances (not all of course) the SEOs are too busy to take the time to explain or train their clients.

    There’s some interesting debate here on the relevance and merit of keyword rankings in an age of localisation, personalisation, “Not provided” and multiple devices – and maybe we should agree to disagree. I’d like to see the debate move on to what alternative metrics should we as SEOs be providing to our clients?

    My starter for 10….

    1. Growth in organic visits (branded vs non-branded)

    2. Growth in sales leads or goal conversions

    3. Growth in sales conversions and revenues

    4. ROI calculations based on the Incremental Growth in Revenues attributable to SEO over the cost of your SEO service

    5. Growth in the number of organic keywords bringing traffic and conversions

    6. Growth in the number of organic landing pages bringing traffic as a percentage of total pages indexed (broken down by branded and non-branded keyword traffic) – in our SEO platform we call this “Organic breadth and depth”

    7. Growth in the number of referring domains that are generating site visits (highlighting the ones that you have done outreach to)

    8. Attribution modelling – what % of sales does organic traffic contribute to across multiple site visits?

    9. Performance of all of the above against targets and budgets set at the beginning of the SEO or inbound marketing campaign

    What else should define the new standard objective metrics that SEOs should be providing to their clients or their bosses?

  • sergiuliano

    Thank Paul, exactly this is the difference many don’t know yet. Local Rankings are overtaking national rankings. Take a look to the next report and you’ll understand exactly why Local Rankings Reports are so important:

    http://goo.gl/m7Bu5

    You could rank on 1st Google page on New York , but not on Google 100 on Miami, for the same Keyword.

    That’s why country rankings only are totally inaccurate

    You should use HeatMap reports for easy spotting each competitor’s position

  • http://www.facebook.com/mikezielonka Mike Zielonka

    Here is the big question: Is anyone using the adwords data in Raven? In all the chatter I have read the argument has been around SERP…but I have yet to read anyone stating that they actually use the Adwords Data.

    I personally do not use it at all and value SERP much more because they are a indicator of progress to me. We know that higher position in SERP leads to more clicks and in this age the exact position you rank doesn’t matter but if you are in the top 1/3, 1/2, etc on page 1 does.

  • Henley Wing

    Why shame on Google? It’s amusing that if you willingly give away data you own, even if you don’t have to.. and then take it away, people make a commotion. Google never had to allow scraping, and all those years where they didn’t enforce the TOS were just them being lenient and giving ppl a free pass.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide