More Info On Google’s Unavailable After Meta Tag & New X-Robots-Tag In Header Support

Last week we reported that Google was to add an “Unavailable After” META Tag. Since then, we’ve spoke to Dan Crow of Google, who provided more information on how to use it, as well information on a new way to send robots blocking info within HTTP headers.

The “unavailable_after” Meta tag will allow you to tell Google that a page should expire from the search results at a specific time. For example, if you have a page that you would like to be removed from the search results at 6pm EST on July 23, 2007, you would add the following Meta tag:

<META NAME=”GOOGLEBOT” CONTENT=”unavailable_after: 23-Jul-2007 18:00:00 EST”>

Once Google crawls the page and sees this Meta tag, it will take about a day for the page to be removed from the search results. Also this tag is only supported with web search. To remove the page completely from Google, including the cache copies, use the Google removal tool.

Google has also added support to control access to non-HTML documents that can’t have meta tags in them for blocking, such as PDF files, audio, xls documents and so. This is through a new X-Robots-Tag directive issued via the HTTP Header used to serve the file.

Here are examples of how to use the X-Robots-Tag:

Don’t index this document:

X-Robots-Tag: noindex

Don’t show cache or snippet in search results for this file:

X-Robots-Tag: noarchive, nosnippet

Don’t index after a specific date (i.e. the “unavailable_after” tag):

X-Robots-Tag: unavailable_after: 7 Jul 2007 16:30:00 GMT

You can also combine these tags as well.

Danny has more on meta tags in Meta Robots Tag 101: Blocking Spiders, Cached Pages & More.

Related Topics: Channel: SEO | Google: SEO | SEO: Blocking Spiders


About The Author: is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here. For more background information on Barry, see his full bio over here.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide