Everything You Wanted To Know About Blocking Search Engines

Jun 12, 2008 at 10:14am ET by Danny Sullivan

Last week, the three major search engines came together to say how they agree — and disagree — over the Robots Exclusion Protocol. It’s such an important standard, one every webmaster should understand. To help, Vanessa Fox has compiled an extensive and outstanding overview of it at Jane & Robot in her Managing Robot’s Access To Your Website post.

The tutorial takes you through key areas such as:

A nice chart showing what you can block using either robots.txt or the meta robots tag for each major search engine. It also covers other things like reverse DNS lookup to verify a crawler’s identity.
Types of content you want private from search engines versus public. Rather than private versus public, "not listed" versus "listed" might be better terms Anything that really should be private ought to be kept behind a password barrier. The tutorial does cover this, but it’s worth stressing that no one should think robots exclusion is a method to keep private/personally identifiable information out of search engines. But there’s other info that you might want "private" in terms of not being listed, such as printer-friendly pages, as the tutorial also explains.
How to block search engines, such as on a site-wide basis using robots.txt, along with tips like using wildcards, specifying particular search engines by crawler name. Page level blocking (with meta tags) is also covered. There are lots of examples.
Common mistakes and myths are addressed, such as the idea that using nofollow alone will keep pages from being indexed. Methods of testing implementation are also covered.

Bookmark the guide — it’s one you’ll want to come back to time and again.

Danny Sullivan is editor-in-chief of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also oversees Search Engine Land’s SMX: Search Marketing Expo conference series, maintains a personal blog called Daggle and can be followed on Twitter here.

See more articles by Danny Sullivan >

Share, Bookmark & Discuss This Article
More:

See more stories like this in the Members Library! Check out the SEO: Blocking Spiders sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!

FREE DAILY SEARCH NEWS RECAP!

Stay on top of all the search news with our daily summary, the SearchCap newsletter. View a sample ›

STAY CURRENT THROUGHOUT THE DAY

The Search Engine Land feed keeps you informed as news happens. SEE ALL FEEDS »

Everything You Wanted To Know About Blocking Search Engines

FREE DAILY SEARCH NEWS RECAP!

STAY CURRENT THROUGHOUT THE DAY

Get Your Search Engine Land Premium Membership!

Get Your Search Engine Land
Premium Membership!