Comments on: Meta Robots Tag 101: Blocking Spiders, Cached Pages & More

By: Igor

Igor — Mon, 07 May 2007 20:54:02 +0000

Microsoft’s Livesearch is ignoring the NOINDEX tag on my pages. I have been e-mailing back and forth with them for a few weeks and things just get curiouser and cruriouser. If you do this search for pages at my site -
http://search.msn.com/results.aspx?q=www.gadgetduck.com&form=QBRE
- you’ll see some pages with a link labeled “cached page”. Those pages carry the NOINDEX metatag. (Also, those pages carry NOARCHIVE. Livesearch actually does not archive the pages – the “cached page” link comes up blank, which is fine with me. But, this seems to show another system glitch for them.)

Has anyone else seen this problem? Is it possible that my tags are malformed? Google and Yahoo seem to have no problem following them correctly.

By: Robert_Charlton

Robert_Charlton — Sat, 10 Mar 2007 20:40:35 +0000

Danny – Another comment… related to my comment above, and possibly something to mention if you talk to responsible people at MSN.

In the past several days, a client site has been plagued by MSN indexing (and prominently ranking) url-only listings to “blocked” pages. These pages have all used the robots meta tag, with the following syntax…

They include AdWords landing pages, test pages, etc. In the past, I’ve encountered problems with Google indexing such references, but that is because the pages were blocked by robots.txt and the “link references” to these pages were exposed. That’s why I ultimately switched to using the robots meta tag and dropped the robots.txt.

As we’ve discussed at SEWF, all the engines need to get on the same page (so to speak) about this.

By: Robert_Charlton

Robert_Charlton — Wed, 07 Mar 2007 19:20:30 +0000

How does the noindex command on AdWords landing pages affect AdsBot and quality score spidering? Google indicates that it will ignore robots.txt so it can spider these pages, but will it spider them if you’re using the robots meta tag?

There are many reasons, I feel, why using the robots meta to block indexing of a landing page might be preferable.

By: Danny Sullivan

Danny Sullivan — Tue, 06 Mar 2007 22:05:26 +0000

Thanks, Sean, much appreciated and have it updated.

By: Sean Carlos

Sean Carlos — Tue, 06 Mar 2007 19:48:19 +0000

As always, a very informative resource.

Just a quick comment to note that Microsoft does support “noarchive”, so the Microsoft only “nocache” is not needed.

Reference:

http://search.live.com/docs/siteowner.aspx?t=SEARCH_WEBMASTER_REF_RestrictAccessToSite.htm

- Sean Carlos

By: Danny Sullivan

Danny Sullivan — Tue, 06 Mar 2007 17:40:44 +0000

You can safely leave it in :)

By: pratt

pratt — Tue, 06 Mar 2007 16:23:40 +0000

I completely agree that it is silly to put:
meta name=”robots” content=”index,follow” in your code. But if it is already in there, are there any benefits to going in and taking it out? Will it speed up load times or anything like that?

Please excuse my lack of programming knowledge.

By: Danny Sullivan

Danny Sullivan — Tue, 06 Mar 2007 10:38:20 +0000

Thanks for the catch! Got it fixed.

By: kching

kching — Tue, 06 Mar 2007 08:15:18 +0000

maybe I haven’t had enough coffee :)
but don’t you mean in you’re second example?
meta name=”robots” content=”index,follow“