• http://www.esnips.com/web/WebAnalyticsGraphs Daniel Waisberg

    Should I understand from this post that the only kind of pages you recommend to block access to crawlers are private, scripts, and duplicate pages? Or these are just examples?

    Do you also believe that internal search pages should be blocked? I understand that there is some controversy on the subject…

    Thank you.
    Daniel Waisberg

  • http://searchengineland.com Danny Sullivan

    Yes, Daniel, I think that was the point of the illustration — to be a general guide of examples as to what someone might block rather than explicit instructions saying you must do this.

    As for search pages, you are correct. Google warns that these should be blocked.

  • http://www.golfvacationinsider.com GolfVacationInsider.com

    Is is possible to block a portion of a page, or only the entire page?

    Thanks in advance for any insight you can provide.

  • http://searchengineland.com Danny Sullivan

    Yahoo supports blocking of parts of pages. See Yahoo Supports New Robots-Nocontent Tag To Block Indexing Within A Page for more about this.