Everything You Wanted To Know About Blocking Search Engines


Last week, the three major search engines came together to say how they agree — and disagree — over the Robots Exclusion Protocol. It’s such an important standard, one every webmaster should understand. To help, Vanessa Fox has compiled an extensive and outstanding overview of it at Jane & Robot in her Managing Robot’s Access To Your Website post.

The tutorial takes you through key areas such as:

  • A nice chart showing what you can block using either robots.txt or the meta robots tag for each major search engine. It also covers other things like reverse DNS lookup to verify a crawler’s identity.
     
  • Types of content you want private from search engines versus public. Rather than private versus public, "not listed" versus "listed" might be better terms Anything that really should be private ought to be kept behind a password barrier. The tutorial does cover this, but it’s worth stressing that no one should think robots exclusion is a method to keep private/personally identifiable information out of search engines. But there’s other info that you might want "private" in terms of not being listed, such as printer-friendly pages, as the tutorial also explains.
     
  • How to block search engines, such as on a site-wide basis using robots.txt, along with tips like using wildcards, specifying particular search engines by crawler name. Page level blocking (with meta tags) is also covered. There are lots of examples.
     
  • Common mistakes and myths are addressed, such as the idea that using nofollow alone will keep pages from being indexed. Methods of testing implementation are also covered.

Bookmark the guide — it’s one you’ll want to come back to time and again.



Danny Sullivan is editor-in-chief of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also oversees Search Engine Land’s SMX: Search Marketing Expo conference series, maintains a personal blog called Daggle and can be followed on Twitter here.

See more articles by Danny Sullivan >


Share, Bookmark & Discuss This Article
More:


Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter


See more stories like this in the Members Library! Check out the SEO: Blocking Spiders sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!

Comments are closed.


RECENT COMMNENTS

  • Buy Advertising said " I've been experimenting with the merger of advertising and entertainment. I think that it can be bot"
  • nickstamoulis said " Wow, this is very interesting, I was not aware of the the Google Books case at all, I will be sure t"
  • nickstamoulis said " These are all very cool, my personal favorite 4th logo is the Ask.com layout, it is very creative!"

See All »


FREE DAILY SEARCH NEWS RECAP!

Stay on top of all the search news with our daily summary, the SearchCap newsletter. View a sample ›

STAY CURRENT THROUGHOUT THE DAY

RSS Feeds

The Search Engine Land feed keeps you informed as news happens. SEE ALL FEEDS »

Upcoming Search Engine Land Conferences

Advertise With Us »

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.


SMX Web Site » | SMX Difference » | SMX News »


Join us at an upcoming SMX event:

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:


See more webcast topics »

TRACK US SOCIALLY
Upcoming Search Engine Land Conferences

Get Your Search Engine Land
Premium Membership!

Become a premium member today and receive:

  • Express commenting privileges & photo.
  • Exclusive videos & newsletters.
  • Discounts to our SMX conferences.
  • Access to "How To" & Other Archives.

Learn More

Upcoming Search Engine Land Conferences
Add to GoogleAdd to My Yahoo!Add to BloglinesAdd to NetvibesAdd to Windows Live