The Robots Exclusion Protocol (REP) is not exactly a complicated protocol and its uses are fairly limited, and thus it’s usually given short shrift by SEOs. Yet there’s a lot more to it than you might think. Robots.txt has been with us for over 14 years, but how many of us knew that in addition to the disallow directive there’s a noindex directive that Googlebot obeys? That noindexed pages don’t end up in the index but disallowed pages do, and the latter can show up in the search results (albeit with less information since the spiders can’t see the page content)? That disallowed pages still accumulate PageRank? That robots.txt can accept a limited form of pattern matching? That, because of that last feature, you can selectively disallow not just directories but also particular filetypes (well, file extensions to be more exact)? That a robots.txt disallowed page can’t be accessed by the spiders, so they can’t read and obey a meta robots tag contained within the page?

A robots.txt file provides critical information for search engine spiders that crawl the web. Before these bots (does anyone say the full word “robots” anymore?) access pages of a site, they check to see if a robots.txt file exists. Doing so makes crawling the web more efficient, because the robots.txt file keeps the bots from accessing certain pages that should not be indexed by the search engines.

Having a robots.txt file is a best practice. Even just for the simple reason that some metrics programs will interpret the 404 response to the request for a missing robots.txt file as an error, which could result in erroneous performance reporting. But what goes in that robots.txt file? That’s the crux of it.

Both robots.txt and robots meta tags rely on cooperation from the robots, and are by no means guaranteed to work for every bot. If you need stronger protection from unscrupulous robots and other agents, you should use alternative methods such as password protection. Too many times I’ve seen webmasters naively place sensitive URLs such as administrative areas in robots.txt. You better believe robots.txt is one of the hacker’s first ports of call—to see where they should break into.

Robots.txt works well for:

• Barring crawlers from non-public parts of your website
• Barring search engines from trying to index scripts, utilities, or other types of code
• Avoiding the indexation of duplicate content on a website, such as “print” versions of html pages
• Auto-discovery of XML Sitemaps

At the risk of being Captain Obvious, the robots.txt file must reside in the root of the domain and must be named “robots.txt” (all lowercase). A robots.txt file located in a subdirectory isn’t valid, as bots only check for this file in the root of the domain.

Creating a robots.txt file is easy. You can create a robots.txt file in any text editor. It should be an ASCII-encoded text file, not an HTML file.

Robots.txt syntax

• User-Agent: the robot the following rule applies to (e.g. “Googlebot,” etc.)
• Disallow: the pages you want to block the bots from accessing (as many disallow lines as needed)
• Noindex: the pages you want a search engine to block AND not index (or de-index if previously indexed). Unofficially supported by Google; unsupported by Yahoo and Live Search.
• Each User-Agent/Disallow group should be separated by a blank line; however no blank lines should exist within a group (between the User-agent line and the last Disallow).
• The hash symbol (#) may be used for comments within a robots.txt file, where everything after # on that line will be ignored. May be used either for whole lines or end of lines.
• Directories and filenames are case-sensitive: “private”, “Private”, and “PRIVATE” are all uniquely different to search engines.

Let’s look at an example robots.txt file. The example below includes:

• The robot called “Googlebot” has nothing disallowed and may go anywhere
• The entire site is closed off to the robot called “msnbot”;
• All robots (other than Googlebot) should not visit the /tmp/ directory or directories or files called /logs, as explained with comments, e.g., tmp.htm, /logs or logs.php.

User-agent: Googlebot
Disallow:

User-agent: msnbot
Disallow: /

# Block all robots from tmp and logs directories
User-agent: *
Disallow: /tmp/
Disallow: /logs # for directories and files called logs

What should be listed on the User-Agent line? A user-agent is the name of a specific search engine robot. You can set an entry to apply to a specific bot (by listing the name) or you can set it to apply to all bots (by listing an asterisk, which acts as a wildcard). An entry that applies to all bots looks like this:

User-Agent: *

Major robots include: Googlebot (Google), Slurp (Yahoo!), msnbot (MSN), and TEOMA (Ask).

Bear in mind that a block of directives specified for the user-agent of Googlebot will be obeyed by Googlebot; but Googlebot will NOT ALSO obey the directives for the user-agent of * (all bots).

What should be listed on the Disallow line? The disallow lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).

Examples:

• To block the entire site: Disallow: /
• To block a directory and everything in it: Disallow: /private_directory/
• To block a page: Disallow: /private_file.html
• To block a page and/or a directory named private: Disallow: /private

If you serve content via both http and https, you’ll need a separate robots.txt file for each of these protocols. For example, to allow robots to index all http pages but no https pages, you’d use the robots.txt files as follows, for your http protocol:

User-agent: *
Disallow:

And for the https protocol:

User-agent: *
Disallow: /

Bots check for the robots.txt file each time they come to a website. The rules in the robots.txt file will be in effect immediately once it is uploaded to the site’s root and the bot comes to the site. How often it is accessed varies on how frequently the bots spider the site based on popularity, authority, and how frequently content is updated. Some sites may be crawled several times a day while others may only be crawled a few times a week. Google Webmaster Central provides a way to see when Googlebot last accessed the robots.txt file.

I’d recommend using the robots.txt analysis tool in Google Webmaster Central to check specific URLs to see if your robots.txt file allows or blocks them, see if Googlebot had trouble parsing any lines in your robots.txt file, and test changes to your robots.txt file.

Some advanced techniques

The major search engines have begun working together to advance the functionality of the robots.txt file. As alluded to above, there are some functions that have been adopted by the major search engines, and not necessarily all of the major engines, that provide for finer control over crawling. As these may be limited though, do exercise caution in their use.

Crawl delay: Some websites may experience high amounts of traffic and would like to slow search engine spiders down to allow for more server resources to meet the demands of regular traffic. Crawl delay is a special directive recognized by Yahoo, Live Search, and Ask that instructs a crawler on the number of seconds to wait between crawling pages:

User-agent: msnbot
Crawl-delay: 5

Pattern matching: At this time, pattern matching appears to be usable by the three majors: Google, Yahoo, and Live Search. The value of pattern matching is considerable. Let’s look first at the most basic of pattern matching, using the asterisk wildcard character. To block access to all subdirectories that begin with “private”:

User-agent: Googlebot
Disallow: /private*/

You can match the end of the string using the dollar sign ($). For example, to block URLs that end with .asp:

User-agent: Googlebot
Disallow: /*.asp$

Unlike the more advanced pattern matching found in regular expressions in Perl and elsewhere, the question mark does not have special powers. So, to block access to all URLs that include a question mark (?), simply use the question mark (no need to “escape” it or precede it with a backslash):

User-agent: *
Disallow: /*?*

To block robots from crawling all files of a specific file type (for example, .gif):

User-agent: *
Disallow: /*.gif$

Here’s a more complicated example. Let’s say your site uses the query string part of the URLs (what follows the “?”) solely for session IDs, and you want to exclude all URLs that contain the dynamic parameter to ensure the bots don’t crawl duplicate pages. But you may want to include any URLs that end with a “?”. Here’s how you’d accomplish that:

User-agent: Slurp
Disallow: /*? # block any URL that includes a ?
Allow: /*?$ # allow any URL that ends in a ?

Allow directive: At this time, the Allow directive appears to only be supported by Google, Yahoo, and Ask. Just as it sounds, it works the opposite of the Disallow directive and provides the ability to specifically call out directories or pages that may be crawled. This may be beneficial after large sections or the entire site has been disallowed.

To allow Googlebot into only the “google” directory:

User-agent: Googlebot
Disallow: /
Allow: /google/

Noindex directive: As mentioned above, this directive offers benefits in eliminating snippetless title-less listings from the search results, but it’s limited to Google. Its syntax exactly mirrors Disallow. In the words of Matt Cutts:

“Google allows a NOINDEX directive in robots.txt and it will completely remove all matching site URLs from Google. (That behavior could change based on this policy discussion, of course, which is why we haven’t talked about it much.)”

Sitemap: An XML sitemap file can tell search engines about all the pages on your site, and optionally, to provide information about those pages, such as which are most important and how often they change. It acts as an auto-discovery mechanism for the spider to find the XML sitemap file. You can tell Google and other search engines about your Sitemap by adding the following line to your robots.txt file:

Sitemap: sitemap_location

The sitemap_location should be the complete URL to the Sitemap, such as: http://www.example.com/sitemap.xml. This directive is independent of the user-agent line, so it doesn’t matter where you place it in your file. All major search engines support the Auto-Discovery Sitemap protocol, including Google, Yahoo, Live Search, and Ask.

While auto-discovery provides a way to inform search engines about the sitemap.xml file, it’s also worthwhile verifying and submitting sitemaps directly to the search engines through each of their webmaster consoles (Google Webmaster Central, Yahoo Site Explorer, Live Search Webmaster Center).

More about Google’s bots

Google uses several different bots (user-agents). The bot for web search is Googlebot. Google’s other bots follow rules you set up for Googlebot, but you can set up additional rules for these specific bots as well. Blocking Googlebot blocks all bots that begin with “Googlebot”.

Here’s a list of Google robots:

• Googlebot: crawls pages from web index and news index
• Googlebot-Mobile: crawls pages for mobile index
• Googlebot-Image: crawls pages for image index
• Mediapartners-Google: crawls pages to determine AdSense content, only crawls sites if show AdSense ads
• Adsbot-Google: crawls to measure AdWords landing page quality, only crawls sites that use Google AdWords to advertise

You can block Googlebot entirely by using:

User-agent: Googlebot
Disallow: /

You can allow Googlebot, but block access to all other bots:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

Issues with robots.txt

Pages you block by using robots.txt disallows may still be in Google’s index and appear in the search results — especially if other sites link to them. Granted, a high ranking is pretty unlikely since Google can’t “see” the page content; it has very little to go on other than the anchor text of inbound and internal links, and the URL (and the ODP title and description if in ODP/DMOZ.) As a result, the URL of the page and, potentially, other publicly available information can appear in search results. However, no content from your pages will be crawled, indexed or displayed.

To entirely prevent a page from being added to a search engine’s index even if other sites link to it, use a “noindex” robots meta tag and ensure that the page is not disallowed in robots.txt. When spiders crawl the page, it will recognize the “noindex” meta tag and drop the URL from the index.

Robots.txt and robots meta tag conflicts

If the robots.txt file and robots meta tag instructions for a page conflict, bots follow the most restrictive. More specifically:

• If you block a page with robots.txt, bots will never crawl the page and will never read any robots meta tags on the page.
• If you allow a page with robots.txt but block it from being indexed using a robots meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.

While robots.txt files are to protect content on a site from being indexed, including a robots.txt file regardless is recommended as many robotic processes look for them and offering one can only expedite their procedures. Together, robots.txt and robots meta tags give you the flexibility to express complex access policies relatively easily:

• Removing an entire website or part of a website.
• Avoiding indexation of images in Google Image Search and other image engines.
• Avoiding indexation of duplicate content on a site.
• Removing individual pages on a site using a robots Meta tag.
• Removing cached copies and snippets using a robots Meta tag.

Both robots.txt and robots meta tag rely on cooperation from the robots, and are by no means guaranteed to work for every robot. If you need stronger protection from robots and other agents, you should use alternative methods such as password protection.

A month ago, Google, Yahoo and Microsoft announced they will be supporting a new canonical tag that allows you to tell search engines that page X is a duplicate page to page Z. In a way, it is a 301 redirect, without the physical redirect.

The tag is incredibly powerful, as are 301 redirects and using this tag should be done with caution and slowly. Matt Cutts posted a new video explaining how one should go about using this tag, being that it is so new. Here is the video:

The Live Search Blog announced they are letting a new robot loose. The new search engine crawler is named msnbot/2.0b and will be added to the army of current MSN spiders, currently named msnbot/1.1.

The new spider is currently being tested but will ultimately replace the old spider. The new spider will respect the current robots.txt protocol set up for MSNBot, so no need to set up anything new in your robots.txt file. In addition, Microsoft promised to crawl slowly in their msnbot/2.0b tests.

MSNBot/1.1 is not that old. It was added back in February of this year and introduced HTTP compression and conditional gets.

I reported at the Search Engine Roundtable this morning that Google said if your robots.txt is unreachable, your site might not make it into the Google index. By unreachable, Google means that if your server simply times out and does not return any server response when Googlebot attempts to access your robots.txt file, then it might not include any of your pages in their index.

Googler John Mueller explained that Google tends to lean on the “safe” side when this situation pops up. When I showed this to Danny, he felt it was ironic that if Google can’t read what you want to block, it might block everything. But if you think about it, with all the legal woos Google has to deal with about indexing content, should they risk indexing a site that might have a nofollow directive in their robots.txt file?

It is important to clarify that a robots.txt file is not required in order to be listed with Google. If you don’t have one and Google sees a normal server status response such as a 404 not found, all’s good. It’s only if Google asks for a robots.txt file and gets no response at all where this might be an issue. Rare case, but good to know.

Last week, the three major search engines came together to say how they agree — and disagree — over the Robots Exclusion Protocol. It’s such an important standard, one every webmaster should understand. To help, Vanessa Fox has compiled an extensive and outstanding overview of it at Jane & Robot in her Managing Robot’s Access To Your Website post.

The tutorial takes you through key areas such as:

• A nice chart showing what you can block using either robots.txt or the meta robots tag for each major search engine. It also covers other things like reverse DNS lookup to verify a crawler’s identity.
   
• Types of content you want private from search engines versus public. Rather than private versus public, "not listed" versus "listed" might be better terms Anything that really should be private ought to be kept behind a password barrier. The tutorial does cover this, but it’s worth stressing that no one should think robots exclusion is a method to keep private/personally identifiable information out of search engines. But there’s other info that you might want "private" in terms of not being listed, such as printer-friendly pages, as the tutorial also explains.
   
• How to block search engines, such as on a site-wide basis using robots.txt, along with tips like using wildcards, specifying particular search engines by crawler name. Page level blocking (with meta tags) is also covered. There are lots of examples.
   
• Common mistakes and myths are addressed, such as the idea that using nofollow alone will keep pages from being indexed. Methods of testing implementation are also covered.

Bookmark the guide — it’s one you’ll want to come back to time and again.

Today, Google, Yahoo!, and Microsoft have come together to post details of how each of them support robots.txt and the robots meta tag. While their posts use terms like “collaboration” and “working together,” they haven’t joined together to implement a new standard (as they did with sitemaps.org). Rather, they are simply making a joint stand in messaging that robots.txt is the standard way of blocking search engine robot access to web sites. They have identified a core set of robots.txt and robots meta tag directives that all three engines support:

Google and Yahoo! already supported and documented each of the core directives, and Microsoft supported most of them before this announcement. In their posts, they also list the directives they support that may not be supported by the other engines.

For robots.txt, they all support:

• Disallow
• Allow
• Use of wildcards
• Sitemap location

For robots meta tags, they all support:

• noindex
• nofollow
• noarchive
• nosnippet
• noodpt

With this announcement, Microsoft appears to be adding support for the use of * wildcards (which will go live later this month) and the Allow directive. The biggest discrepancy is with the crawl-delay directive. Yahoo! and Microsoft support it, while Google does not (although Google does support control of crawl speed via Webmaster Tools ).

This isn’t the first time the major search engines have come together for an announcement regarding how they support publishers. In late 2006, all three joined together to support XML Sitemaps and launched sitemaps.org, followed in April 2007 with support for Sitemaps autodiscovery in robots.txt, and in February 2008 with more support for more flexible storage locations of Sitemap files. In early 2005, the engines declared support for the nofollow attribute on links (in an effort to combat comment spam).

Why are the search engines coming together to talk about their varied support for traditional methods for blocking access to web content? A Microsoft spokesperson told me that while robots.txt has been the de facto standard for some time, the search engines had never come together to detail how they support it and said the aim is to “make REP more intuitive and friendly to even more publishers on the web.” Google similarly said that “doing a joint post allows webmasters to see how we all honor REP directives, the majority of which are identical, but we also call out those that are not used by all of us.”

Yahoo! told me:

Our goal is to come out with clear information about the actual support around REP for all engines. We have all separately at different times reported our support and this creates a long trail hard for anyone to put together. Posting the same spec at the same time provides a sync point for everyone as to the actual similarities or differences between our implementations for all engines. We are trying to address the latent concerns around differences across the engines.

Of course, each engine has provided documentation in their respective help centers for some time, and Google and Microsoft provide robots.txt analysis tools that detail how they interpret a file in their webmaster tools, so while they haven’t documented their support jointly, the documentation itself isn’t new.

This move may be an effort to show a consolidated front in light of the ongoing publisher attempts to create new search engine access standards with ACAP. This direction reflects the ongoing direction of the messaging the search engines have had about ACAP. For instance, Rob Jonas, Google’s head of media and publishing partnerships in Europe, said in March that “the general view is that the robots.txt protocol provides everything that most publishers need to do.”

For more information, see each engine’s blog posts (updated as their posts go live):

• Microsoft Live Seach Webmaster Blog
  
• Google Webmaster Central Blog
• Yahoo! Search Blog

Google’s rolled out a new tool at Google Webmaster Central, a robots.txt generator. It’s designed to allow site owners to easily create a robots.txt file, one of the two main ways (along with the meta robots tag) to prevent search engines from indexing content. Robots.txt generators aren’t new. You can find many of them out there by searching. But this is the first time a major search engine has provided a generator tool of its own.

It’s nice to see the addition. Robots.txt files aren’t complicated to create. You can write them using a text editor such as notepad with just a few simple commands. But they can still be scary or hard for some site owners to contemplate.

To access the tool, log-in to your Google Webmaster Tools account, then click on the Tools menu option on the left-hand side of the screen after you select one of your verified sites. You’ll see a "Generate robots.txt" link among the tool options. That’s what you want.

By default, the tool is designed to let you create a robots.txt file to allow all robots into your site. That’s kind of odd. By default, all robots will come into your site. If you want them, then there’s no need to have a robots.txt file at all. It’s like pinning a note to your chest reminding yourself to breathe. Promise, you’ll keep breathing even if you forget to look at the note.

Instead, you generally want to put up a robots.txt file to block crawling of some type. I may dig into a future article to examine when you might want to mix allow and disallow statements, but off the top of my head, there’s not a lot of reasons to do so.

You can change the default option to "Block all robots" easily enough. Do that, and you get the standard and familiar two line keep out code:

User-Agent: *
Disallow: /

The first line — User-Agent — is how you tell particular spiders or robots to pay attention to the following instructions. Using the wildcard — * — says "hey ALL spiders, listen up."

The second line says what they can’t access. In this case, the / means to not spider anything within the web site. You know how pages within a web site all begin domain/something, like this:

http://website.com/page.html

See that / between website.com and page.html? Technically, that slash is the start of the URL. So if you disallow all pages beginning with a slash, you’re blocking all pages within the entire site.

Let’s move on from our mini-robots.txt 101 course. Maybe you only want to block Google. Well, the tool is supposed to make this type of thing easy, but I was perplexed. Step one is to either allow or block ALL robots. Then in Step 2, you decide if you want to block specific robots. So which do you go with in step 1, block all or none?

I figured you’d want to allow all robots, then believe the reassuring text next to that option that said "you can fine-tune this rule in the next step." The problem is, I couldn’t. If I tried to block Googlebot, the instructions didn’t change. If I tried to choose, say, Googlebot-Mobile, same thing.

Eventually, I figured it out. If you decide to block specific spiders, you have to choose the spider, then specify also what you want to block in the "Files or directories" box, such as a particular file or directory. So say I kept all print-only versions of stories in a directory called /print. I’d enter that directory to get this:

User-Agent: *
Allow: /

User-Agent: Googlebot
Disallow: /print
Allow: /

The first part tells spiders they can access the entire site. As I said, this is entirely unnecessary, but you get it anyway. The second part says that Googlebot cannot access the /print area.

The tool lets you craft specific rules for these particular Google crawlers:

• Googlebot
• Googlebot-Mobile
• Googlebot-Image
• Mediapartners-Google
• Adsbot-Google

I wish the names were accompanied by parenthesis quickly explaining what each crawler does, and what blocking them will do, say, something like this:

• Googlebot-Mobile (allows or blocks content from Google mobile search)

Instead, you have to look through the various help files to understand what each does. Ironically, the older Analyze Robots.txt tool within Google Webmaster Tools DOES have these helpful explanations, so I expect they’ll migrate over.

You can also use the tool to enter a name for another crawler. The problem is, someone using this tool probably doesn’t know the crawler names out there that they want to block. I’d have given Google serious kudos points if they added some of the other major crawlers. But then again, if they had, no doubt someone would have accused them of trying to get people to block other search engines :)

Another thing that would have been nice was if people could have pasted full URLs into the box to have them converted. A site owner using this tool might not realize they need to drop the domain portion of a URL to block a particular page. But if you could paste something like this:

http://website.com/page-i-want-to-block.html

And have the tool automatically turn it into this:

User-Agent: *
Disallow: /page-i-want-to-block.html

After you make your file, upload it to the root directory of your web site. If you don’t know what that is, find someone who does! This is important. Google allows for subdirectories of web sites to be registered within Google Webmaster Tools. However, robots.txt files do NOT work on a subdirectory basis. They have to go at the root level of a web site. If you don’t put them there, then you won’t be preventing access to any part of the site. Remember, after you upload to the root level, you can go back into Google Webmaster Tools and use that aforementioned analysis tool to see if it is really blocking the pages you want to keep out.

Overall, I’m glad to see the new tool, and I imagine it will improve more over time to make it even more user friendly.

In related news, Google says that the Web Crawl diagnostics area now has a new filter letting you see only web crawl errors related to sitemaps you’ve submitted. Also, there have been some UI tweaks to the iGoogle gadgets from Webmaster Central that were rolled out last month.

For more about Google’s webmaster tools, be sure to check out the quick start guide they offer and see our Google Webmaster Central archives.

Matt Cutts of Google posted a blog entry asking SEOs how they want Google to handle the NOINDEX meta tag. If you use the NOINDEX meta tag now, Google won’t show the page in any way in the Google index — not even a “link only” listing.

Matt asks SEOs if this is what they want and the poll currently shows us that yes, SEOs want it this way. Here are the current results, but the results may change over the course of the week:

How should Google treat the NOINDEX meta tag?

• 240 say “Don’t show a page at all.”
• 24 say “Find some middle ground.”
• 23 say “Show a link to the page.”

Google Explains The NOINDEX, NOFOLLOW, NOARCHIVE & NOSNIPPET META Tags from last year has more about the various meta commands Google allows, and Meta Robots Tag 101: Blocking Spiders, Cached Pages & More goes into depth about how each feature is used for particular search engines.

The Yahoo Blog issued a weather report for changes to rankings in Yahoo Search, along with news that they are now supporting the X-Robots-Tag directive — a way to control indexing of content that cannot accept meta robots tags.

Google also supports X-Robots, which gives webmasters the ability to define robots.txt like rules within http headers, as opposed to just the META data within HTML pages.

Yahoo provided a few examples of how it can work:

• X-Robots-Tag: NOINDEX — If you don’t want to show the URL in the Yahoo! Search results. Note: We’ll still need to crawl the page to see and apply the tag, so if you don’t wish to have the page crawled, use robots disallow on robots.txt.
• X-Robots-Tag: NOARCHIVE — If you don’t want to display the cache link in the search results page.
• X-Robots-Tag: NOSNIPPET — If you don’t want to display a summary in the search results page.
• X-Robots-Tag: NOFOLLOW — If you don’t want Yahoo! to crawl links in the page.

After a year of discussions, ACAP — Automated Content Access Protocol — was released today as a sort of robots.txt 2.0 system for telling search engines what they can or can’t include in their listings. However, none of the major search engines support ACAP, and its future remains firmly one of "watch and see." Below, more about the how and why of ACAP.

Let’s start with some history. ACAP got going in September 2006, backed by major European newspaper and publishing groups that in particular felt Google was using content without proper permissions and wanting a more flexible means to provide this than allowed by the long-standing robots.txt and meta robots standards.

These two standards are found at the robotstxt.org, and ACAP has been referring to them often at "Robots Exclusion Protocol" or REP, though within the SEO world, they’re generally known by their actual names.

Robots.txt was born in 1994 as a way to block content on a server-wide basis; meta robots emerged in 1996 as a system to block on a page-by-page basis (see Meta Robots Tag 101: Blocking Spiders, Cached Pages & More for more about it). Neither has been updated since those years ago, in terms of search engines coming together to agree on new universal standards. In short, REP has no "guardians" or group to take it forward.

Enter ACAP. If the search engines weren’t going to improve robots.txt, the aforementioned publishers decided they’d take on the challenge. Of course, creating a standard for search engine indexing is kind of a waste of time, if you don’t have the search engines themselves to actually support it. But ACAP didn’t let that be a deterrent. Over the past year, it has had a working group setting up a new system, with search engines Google and Ask.com, along with Exalead, taking part in the discussions. FYI, I’ve not been an active working member, but I’ve been included on the working group’s emails and chimed in from time to time with advice and thoughts.

The ACAP System

Now the new system has arrived, being unveiled at the ACAP conference in New York today. Before getting into support, let’s cover what’s in it. You’ll find an overview page for the specifications here, which leads to:

• A robots.txt-to-ACAP conversion tool (don’t worry; this should make your robots.txt file still work as a regular one and double as an ACAP file)
   
• ACAP extensions to use with robots.txt (here, PDF file)
   
• ACAP extensions to use with meta robots (here, PDF file)
   
• ACAP logo for those that want to show they’re using ACAP (not required to make ACAP work, but expect publishers pushing ACAP to make use of it)

What does ACAP provide that robots.txt and meta robots does not? After going through the technical specs, which are pretty dense reading, I’d summarize it this way:

• Emphasis on both granting permissions and blocking
   
• Support for time-based inclusion or exclusion

That’s it. Discussions have covered concepts such as how password-protected content could be indexed, or whether you could issue permissions on a country-by-country basis, but some of these ideas haven’t made it into the first cut.

AP has a nice overview article about the ACAP launch, and I found the companion piece a nice summary if you’re looking for some faster specifics. A key part:

Some search engines have interpreted "disallow" to mean that the site cannot be added to the index but could be fetched for use in various algorithms employed to determine how high a site appears in search results….ACAP proposes to clarify that "disallow" refers to indexing.

A separate "crawl" command would be added to bar the indexing software or crawler entirely.

In addition, Web sites would be able to add qualifiers stipulating that the information expires from the search index on a specific date, in a given number of days or whenever the crawler returns to the site.

A "follow" command would permit or block the crawler from following links within a page.

"Preserve," with similar time limits available for "index," would stipulate whether a copy may be stored in a search engine’s cache.

"Present" would govern a search engine’s ability to display the copy, and a site may limit that further — for example, to a snippet or to a miniaturized version, or thumbnail.

As I said, there’s an emphasis on granting permission. By default, search engines assume everything is open to indexing. ACAP changes this assumption, asking those that create the files to explicitly indicate yes or no.

Should You Use It?

So now we have a new standard for expressing search engine permissions. Do site owners need to run out and immediately use it?

No. Not immediately. Not even long term.

Right now, none of the major search engines are supporting ACAP. If you were to use ACAP without ensuring that standard robots.txt or meta robots commands were also included, you’d fail to properly block search engines. Only Exalead, which is not a major multi-country service, would currently act upon your ACAP-only commands.

Even if ACAP were to magically get endorsed and supported by all the major search engines, robots.txt and meta robots support wouldn’t go away for many years. There are simply too many sites that use those systems, have used them for over a decade, and would fail to upgrade. Those two systems will continue to be supported in the same way Microsoft has had to support DOS programs despite the growth of Windows.

So why bother at all? Probably two reasons:

• You want to personally test out how ACAP works, playing with the permissions and seeing what happens in Exalead
   
• You want to support the ACAP system and hope that if enough people use it, perhaps the search engines will adopt it. FYI, ACAP is urging (PDF file) "universal adoption" by publishers by the end of next year.

Search Engine Support

What’s up with the major services? I emailed the big three, Google, Microsoft, and Yahoo, all of whom either took part in the working group or are at today’s conference. Google’s canned answer:

We are interested in all initiatives that allow web publishers and search engines to work more closely together. We have undertaken many efforts in this direction over the years including supporting file-extension and wildcard specifications in robots.txt, SiteMaps, our Webmaster Console, extending per-item indexing specification to non-html documents and specifying how long a url would be available. We will examine ACAP proposals when they become available. As a broad-based search engine, we need to keep in mind the needs of millions of web publishers worldwide.

As it happens, I was at Microsoft yesterday, and while I haven’t gotten a formal statement to post, the sentiment was the same as Google. Microsoft is interesting in supporting publishers, is continuing to grow its own tools and will also watch ACAP, wanting to support publishers in general

Yahoo’s not sent a statement back yet, but when it arrives, you can expect it will be pretty much the same as Google and Microsoft.

Why not just jump into ACAP? Between the lines time here — no one really wants to hand over control of the standard to the ACAP group, especially in my view when it has been born out of some anti-search engine hype.

So why not jump behind improving robots.txt and meta robots? Another issue here is that no one is officially in charge of those standards. The search engines are sort of the gatekeepers, because it’s what they decide to support that effectively becomes "law." If they don’t support a particular exclusion command, it might as well not exist.

The various search engines tell me they have been talking more about making some collective improvements. Individually, they’ve already added to both robots.txt and meta robots over the years, extensions that may work with their particular search engines. Perhaps they will become more unified.

In particular, they’ve united around the sitemaps standard. That sort of picks up what ACAP does in terms of being a system to provide express permission of indexing, and it’s where I’d expect any search engine-driven, collective agreement about improved blocking tools to emerge.

Also be sure to read Up Close & Personal With Robots.txt, which summarizes the second robots.txt summit that I organized earlier this year. The article covers a lot of things that general site owners and SEOs have wished for, along with some search engine responses.

Conclusion

So has the entire ACAP project been a waste of time, or as Andy Beal’s great headline put it when ACAP was announced last year, Publishers to Spend Half Million Dollars on a Robots.txt File? That still makes me laugh.

No, I’d say not. I think it’s been very useful that some group has diligently and carefully tried to explore the issues, and having ACAP lurking at the very least gives the search engines themselves a kick in the butt to work on better standards. Plus, ACAP provides some groundwork they may want to use. Personally, I doubt ACAP will become Robots.txt 2.0 — but I suspect elements of ACAP will flow into that new version or a successor.