• Bryan Ceguerra

    This one, is a good example that Google is not de-indexing links.

  • aaronrobb

    And how would you remove the page fully if you can’t control the links?

  • http://profiles.google.com/franklyn54 Franklyn Galusha

     We had this very same problem with a new client who for one reason or another did not want certain pages on his website to show up in the results.  Blocking Google’s little robot is more challenging than you might think.

  • w3origin

    Its a serious issue, what is the best way to resolve it ?

  • http://twitter.com/YoungbloodJoe Joe Youngblood

    make a robots noindex,nofollow meta tag, let the page be indexed. easy cheesy breezy beautiful

  • http://twitter.com/YoungbloodJoe Joe Youngblood

    google stopped obeying robots.txt last year. this is nothing new, they are just displaying a new message. if the page gets linked or posted to social somewhere gbot might find and index it. if you block gbot it can’t follow index directives. I dont think the title came from the page either, notice it’s not the same title as on the page or even teh same H1. Looks like a link Anchor or surrounding text was used.

  • http://twitter.com/BrewSEO BrewSEO

    Also, Google will index (and display) a page that is shared on Google+ regardless of it’s Robots status. They said they need to crawl the page to show it’s preview on Google+ 

  • aaronrobb

    Could that be directory-wide?

    And I think it is breezy, yes! :)

  • http://twitter.com/YoungbloodJoe Joe Youngblood

    unfortunately no. it’s is on a page by page basis. there is no robots.txt noindex /directory command.

    however you can go into webmaster tools and remove the directory. it will get pulled out for 90 days (i think that’s the time frame) and during that time you can add the robots meta tags to the pages. once that is done they will get deindexed.

  • HomeMade14

    Heres a question for ya experts. I am changing my site ( slowly ) from html to php. So, on the older html, i added the no index no follow. The newer php has most of the same content. I thought i would have a problem with duplicute content and maybe have a problem… Any thoughts on what i should do? thanks so much

  • http://twitter.com/jorgepinon Jorge

    This message about robots.txt is difficult to remove too. I removed our robots.txt file days ago and it still appears that way in search results.

    I even used Google’s Webmaster tools to try to get it crawled again. That was 2 days ago.