• vivitron

    “Blocking a URL in robots.txt will not save your “crawl budget” because they still keep the URLs in their index.”

    You have to be careful how you read this – from my experience, adding pages to robots.txt DOES stop Google from crawling the page in the future. Evidence from web server logs and crawl graphs in webmaster tools.

    Any other opinions?

  • http://www.winwinmantra.com winwinmantra

    I totally agree with @vivitron.

    The same sentence can be rewritten in this way.

    Blocking a URL in robots.txt will not save the URL from indexing if Google finds the same URL at any other website. It will index it irrespective of whether is blocked in robots file on actually website.

    So do not use those URLs as inbound linking (backlink) which you have blocked in robots file.

  • http://incrediblehelp incrediblehelp

    So I assume he is talking about one domain to another? Not internal 301 redirects that done very often through design changes?

    What about the Webmaster Tools feature that signals directly to Google that a domain is moving? Is their decay there?

    What about the canonical tag they Google as preached to use. Is their decay there?

    …and when i refer to PR I am talking about the back link credit a certain URL has achieved

    Why does Google say something like this without explaining the full story?