• http://webmd.com roseberry

    Cutts said those are “probably old pages that haven’t been recrawled,” and said they’ll “naturally drop down” as the URLs are recrawled.

    Umm, so Matt’s saying that Google is (re)crawling pages that have a disallow in the robots file? Don’t see how they “naturally” do anything given that the recrawl won’t happen – unless i’m missing something.

  • http://www.webdesign-solutions.blogspot.com/ Daniel Marriott

    i like the sound of the +1 but i think it will be abused to much

  • http://www.michael-martinez.com/ Michael Martinez

    @roseberry, “so Matt’s saying that Google is (re)crawling pages that have a disallow in the robots file? ”

    Doubtful. But you can always test what happens for yourself by disallowing a section of your site for a few months and see what happens.

  • http://webmd.com roseberry

    @michael – yeah that was kind of sarcasm given that it was implied that the pages would “naturally” disappear once they were recrawled and they found the 301, but of course there’s the disallow so they won’t be “(re)crawled” so something else would have to happen for these to be removed from the index. Again, that’s what i’m assuming. Though looks like all these old pages have magically disappeared since yesterday and all of them all at once. I assume it was just “natural”.

    Don’t know that this matters a great deal regardless what happens.