• alexc

    This is probably formatting issue, but anyone new to robots.txt should bear in mind that each directive should start on new line, ie:

    User-agent: Googlebot
    Disallow:

    If you just copy/paste directives as shown above you’d have incorrect disallow directives that won’t work.

    “The rules in the robots.txt file will be in effect immediately once it is uploaded to the site’s root and the bot comes to the site.”

    The rules will only be in effect when bots take them and interprete – this won’t happen immediately after change is made and this delay will depend on a search engine: it can easily be days.

  • franjeanes

    I was doing some research on robots.txt and found your very informative post – thank you.

    My initial search query surprised me and I ended up writing a blog post about it prompting me to think about how we use, or sometimes forget to use a robots.txt file in the most beneficial way.