• Ian Lurie

    I got some great feedback from other SEOs right before this article went live, and wanted to put some extra info in a comment:

    1. I talk about crawl budget a lot, but the most important part of this article, really, is ‘making the most of your crawl budget’. That’s what Vanessa Fox refers to as ‘crawl efficiency’. If you have a huge crawl budget but 90% of it is sucked up by duplicate content and such, then you won’t see much benefit.

    2. Search engines apply PageRank to individual pages, not to entire sites. However, I definitely see evidence that they look at some version of average PageRank, or domain authority, or whatever you want to call it, site-wide. That appears to impact crawl budget. There may not be a causal link – it may simply be that sites with lots of pages that all have higher PageRank get higher crawl budgets as a side effect of deep links to individual site pages.

    This is a very deep, geeky topic – I hope I don’t ruffle any feathers too badly, but I also hope that a lot of people smarter than I will pick up the discussion, so we can all learn something.



  • http://www.linkedin.com/in/brianrbrown Brian R. Brown

    Ian, great article…crawl equity is one of my favorite geeky search topics! Nice analysis and I agree pretty much across the board….however, #8 around faceted navigation can be a slippery slope. Like many things related to SEO, faceted or guided navigation is neither good nor bad for SEO (or crawl equity), but rather how it is employed can greatly influence this.

    What I think you were driving at was SEO-friendly guided navigation that helps provide high-value, contextually rich content pages, while also reducing levels of duplication and pagination. Bad faceted navigation can of course create a near infinite crawl path, killing crawl equity.

  • Ian Lurie

    Hi Brian,

    Thanks! I totally agree on navigation. Obviously, if you have a beautiful hammer and just use it to squash your thumb, it doesn’t do a lot of good. Use faceted navigation the way you use any other tool – wisely.


  • http://www.michael-martinez.com/ Michael Martinez

    The is a good first step in an interesting direction but you’ll need to take (number of pages on site), (number of pages crawled), and (number of pages indexed), and/or (number of pages receiving search traffic) into consideration before you can provide any meaningful correlations.

  • Ian Lurie

    Plus, to really make it work, we need a much larger sampling. If anyone wants to volunteer their log files…

  • http://www.jaankanellis.com Jaan Kanellis

    Ian for page authority did you use the home page number from each SEO tool?

  • Ian Lurie

    Jaan I actually averaged PageRank across the top 20% of pages on a given domain.

  • http://www.linkedin.com/in/matteosutto matteo sutto

    Great article Ian

    Regarding your correlation analysis between page crawled and PR, wouldn’t be more meaningful to compare the ratio “page crawled by Google to total pages existing on a domain” to the PR of the site ?
    I suspect there might be a correlation between the PR of a site and the total number of pages on a domain, thus biasing the correlation between page crawled and PR

  • http://www.3sMarketingTeam.com Al

    Wouldn’t a blog post be considered a thin page and as a result not rank well, although blog posts are notorious for ranking easily.

  • Ian Lurie

    Matteo – You’re describing crawl efficiency, which is what I talk about in the ‘making the most’ section. It really depends more on how you use the crawl budget you’ve got.

    It’s POSSIBLE that I’ve got the causation backwards, but I don’t think so. Testing with a bigger sample will prove it at some point.

  • Ian Lurie

    Al – I don’t think so. Blog posts tend to be fairly content-heavy, actually. And a well-tended blog will grow and accumulate links continuously, so if anything, a good blog would perform far better over time.

  • http://alexavery.com.au/seo-ppc-analytics-services/ Alex Avery

    If that’s “Nerd Alert” material, then ring my bell Ian.

    Like many of the comments (are trying to) say above, don’t be afraid to go deeper or get Nerdier. That’s what we’re here for. :)

    Keep up the good work.

  • http://www.greenlaneseo.com/blog Bill Sebald

    Cool article.

    “Crawl budget does not equal lots of organic traffic. It doesn’t equal lots of indexed pages. It merely equals Googlebot visiting many URLs on your site.”

    Inherently it may not guarantee any of that, but for some (especially webmasters/SEOs who like to tinker/add onto existing content on large sites like me), it can lead to a wider net of new rankings, newfound authority, and circle back to more PR and gasoline in the spiders’ tank. For me, That’s more than enough of a reason to focus on it. Personally I don’t need an exhaustive experiment.