• http://www.freshsupercool.com/ FreshSuperCool

    I have a perfect example. I was doing lead generation for solar (photovoltaic) panel sales companies and I would come across people typing “solar system”.. which made perfect sense.

    Who would’ve thought.. but apparently people were looking for solar system as in stuff in the sky. When I took out broad match for solar system, the conversions sucked. But when placed in for exact match, it converted well.

    Who woulda thunk it.

  • Pat Grady

    “we are prone to creating data correlations where none exist.”
    Amen!

    Not just in SEO ranking (front end), horrid damage is done when it comes to attribution (back-end, results),

  • MRubenzahl

    Excellent article, Warren, on a topic not often discussed. The era of “big data” is, I think, encouraging people to see false patterns in mass quantity.

    One strategy I’ve used since long before data was cheap and plentiful was to ask for expected actions before seeking data. Simple example: A business manager asks, “how many visitors clicked on x,” and I ask, “based on the result, what action will you take?” If they have no answer, it is time to rethink the question. Most common are requests to survey. “We want to survey customers who visit this page and ask (whatever).” Most of the time, they could not identify an expected insight, a business result, or an action — they just wanted to know.

  • Colin Guidi

    Solid article. Enjoyed the thought process very much.

  • Claaarky

    Sorry, I’m confused about what the advice is here.

    If I poke you in the eye and you say ‘ouch’, then I do it again with the same outcome, and repeat this over and over with the same outcome, I shouldn’t assume the cause of you saying ‘ouch’ is me poking you in the eye, because my mind wants to see a pattern and therefore could be misleading me (Type 1 error), because the ‘ouch’ that appeared to correlate with me poking you in the eye could coincide with someone else poking you in the ribs at exactly the same time, every time.

    But I also shouldn’t assume I’m wrong to think the cause of you saying ‘ouch’ is me poking you in the eye because I could be overlooking the actual cause (Type 2 error).

    My approach would normally be to take any patterns I’ve found and experiment to see if I can repeat X by doing Y.

    If my data shows a clear correlation, I’ve managed to control the experiment to ensure nobody is poking you in the ribs while I’m poking you the eye (or I’m confident the probability of someone poking you in the ribs at the exact same time as I poked you in the eye is so low that it can be dismissed as a potential alternative cause) can I now be sure I’m not making a Type 1 or 2 error, or do I need more data, or more experiments.

    When do I stop, when do I know I’ve found the real cause, at which point can I be sure some clever dick won’t smugly step in and say “ah but correlation is not causation” without spending a second studying my methods or data (which is an annoying pattern I’ve noticed with SEO experts).

  • Ilya Vareshnuk

    Firstly, I want to say that the idea expressed in the article is great and it certainly deserves to be heard.

    BUT I’m affraid I kind of found a mistake in your very fundamental statements. You said: “we believe a pattern is real when it is not (Type 1), or we don’t believe a pattern is real when it is (Type 2)” but it is totally opposite to what wikipedia (and others sources) say: “a type I error is the incorrect rejection of a true null hypothesis and a type II error is the failure to reject a false null hypothesis”. Luckily it does not affect the whole idea but it confused me a litle bit.

    To be honest I’m not sure even now that it is your mistake, maybe I just misunderstood your point, so it would be great to hear your comment.

  • Ilya Vareshnuk

    Great example! And I agree with your point – if you are sure that “nobody is poking you in the ribs while I’m poking you the eye” than such “correlation IS causation”.

    But how I understood Warren’s point is that you should take all this “patternicity” and cognitive errors into consideration only if you’re not sure in all the factors influence.

  • http://www.warrenlee.org/ Warren Lee

    Thanks, I think you get the point indeed. I am recommending that when you use a pattern based approach to deal with uncertainty you should indeed consider whether or not your making a type 1 or type 2 error. Meaning, you see a pattern when one doesn’t exist, or you fail to recognize the true correlation. Which is why, it takes creativity, looking at all the angles, and testing for statistical significance. Also, I should add that in dealing with ambiguous scenarios I also like to perform what I call a “what if I am wrong & there is no correlation” risk assessment analysis, and also I will estimate a “what if I am right & there is a correlation” impact analysis. This also helps to weigh the pros and cons associated with decisions based on pattern seeking.

  • http://www.warrenlee.org/ Warren Lee

    Sorry for the delayed reply. Your “normal approach” would work for me. If you can repeat X by doing Y, then problem solved, stop there.

  • http://www.warrenlee.org/ Warren Lee

    Hi Illya, I recommend that you do not get hung up on the technical
    definition of a type 1 or type 2 error. The technical definition from
    wikipedia that you have provided appears to be confusing the difference
    between the two for you. The point is that these are both errors in
    judgement that can be made when making pattern based inferences. I am
    not mistaken in the fundamental statement. If you Google “wikipedia type
    1 error” and look at the non technical definition, you will see that a
    type one error in laymans terms according to the same wikipedia page is
    as follows: “Usually a type I error leads one to conclude that a
    supposed effect or relationship exists when in fact it doesn’t.” And in
    this article i’m saying the same thing: “…we believe a pattern is real
    when it is not (Type 1)”. From your reply to the comment by Claaarky, I
    think you do indeed get the gist of the article.

  • http://www.warrenlee.org/ Warren Lee

    In my opinion the link is not as great as many assume. There is
    correlation, but this is not indicative of a direct causal impact on
    rank. Rather there may be more of an indirect causal relationship, and I
    agree with your inclination that this would be of marginal impact. In
    other words, it is a mistake to believe that the value of social media
    on SEO is to get more likes and +1 to improve rankings because whether
    or not there is a causal relationship this misses the point of social
    media’s true value towards improving SEO performance. There is
    significantly more value from social media on SEO when we consider that
    instead it is more valuable to use social media to inform content
    strategy. Within social media our audience is sharing what content
    resonates with them most, and where they want to consume and share it.
    This is content strategy gold! And when marketers apply social listening
    techniques to inform content strategy, they get higher engagement,
    which yields higher social engagement which indirectly helps with SEO.

  • Paul Rone-Clarke

    I absolutely agree. Social engagement is for the benefit of the users and for the website owner to open up other traffic channels. It’s influence over SERP’s is likely a spike – one we can already see being flattened as each update passes.

  • Ilya Vareshnuk

    Thank you for explanation. It seems that I interpreted the definition too much literally. And I did get the gist and trying to keep your idea in mind making the decisions on something unevident.