• http://www.acsius.com/ Arun Singh

    I am wondering why bothering blocking the summary of a page we can all read…

  • http://www.archiewatt.com/ Archie Watt

    Don’t know if I’m missing something, but it looks to me like all petitions are blocked like that regardless of how many signatures it has – look at https://petitions.whitehouse.gov/petitions/popular/0/2/0 – it’s the same with all of those, and several are well over the signature threshold. I can’t see a single petition that *isn’t* blocked by robots.txt.

  • Graham Ginsberg

    If the page or pages are blocking Google, a good move indeed, what gives Google the right to index the page at all and even show it in search results?

    Seems like Google is breaking in and entering websites without permission. Isn’t that a crime in America?

  • http://www.ninebyblue.com Vanessa Fox

    The petitions are blocked. Once a petition reaches a certain number of signatures, a response page for that petition is created that lives under the /responses folder (that contains info on the petition and the response and links back to the original petition). That folder is not blocked.

  • http://www.harisbacic.com/ Haris

    Umm no.

    1. Using robots.txt file does not legally prohibit search engine bots from crawling your websites. It is up to each search engine bot to respect robots.txt rules, but they are absolutely not legally required to do so.

    2. Did you read this part: “Google can guess at what the page is about from other pages linking to it to form a title”? Google did not index the page by visiting it. They did so by creating a guess of what the page is about from other websites linking to it.

  • StevenTMahlberg

    Maybe it’s just because Google is a tool of the Government and they don’t want to deal with stupid petitions.