Yahoo Provides NOYDIR Opt-Out Of Yahoo Directory Titles & Descriptions

Feb 28, 2007 at 2:06pm ET by Danny Sullivan

Yahoo! Search Support for ‘NOYDIR’ Meta Tags and Weather Update from the Yahoo Search Blog covers how at long last, you can now tell Yahoo to not use Yahoo Directory information to make a title and/or description for your web page listings. It also cover how Yahoo’s currently doing a reindexing change that might impact rankings. More on that below, plus tips about also blocking the Open Directory information from being used for your pages and some possible conflicts with multiple robots tags.

Sometimes pages are listed in both Yahoo’s crawler-based search results and within its human-compiled directory, the Yahoo Directory. In those cases, Yahoo usually replaces the title and description of a web page in the crawler-based results with the information from the Yahoo Directory. Yahoo has operated this way for as long as I can remember — that’s over a decade :)

Now this has changed. Sometimes a site owner might not want the Yahoo Directory description to be used for their page. A case in point is Tony Knowles. The Yahoo Directory lists him this way:

Knowles, Tony (D)
Democratic candidate from Alaska for U.S. Senate, 2004.
www.tonyknowles.com

So when you search for him in Yahoo web search service like this – tony knowles — his listing comes up as so:

Tony Knowles
Democratic candidate from Alaska for U.S. Senate, 2004
Category: Alaska > 2004 U.S. Senate Election
www.tonyknowles.com – 15k - Cached - More from this site

The problem is, Yahoo’s directory information outdated. Knowles did run for Senate in 2004, but then he ran for governor of Alaska in 2006 using the same web site. During his governor campaign, this caused Yahoo to list him as a senate candidate, as I covered last year.

Now there’s a solution. By inserting this meta tag on any page:

<META NAME="ROBOTS" CONTENT="NOYDIR">

You can ensure that Yahoo will not use a directory description it might have for that page.

<META NAME="Slurp" CONTENT="NOYDIR">

The first tag tells any spider that wants to recognize the tag not to use a Yahoo Directory title or description. Of course, no other spiders do that — but Yahoo’s just building in some protection should that come up in the future. The second tag specifically tells the Yahoo spider not to use the information.

A related tag is the NOODP meta tag. Similar to how Yahoo works, various search engines have looked at the Open Directory Project to get information to make titles and descriptions of pages they also crawl. Pressure started building back in 2005 that site owners should be able to prevent Open Directory information from being used on their pages in this way. Last year, we saw all the major search engines do it except Ask. Instructions and dates of implementation for each are below:

That solved the ODP issue, but pressure remained on Yahoo to provide a fix for its own directory. Now that’s happened.

Back to the ODP. The tags to use to block the ODP are similar. Here’s the one to use for all spiders:

<META NAME="ROBOTS" CONTENT="NOODP">

And if you wanted to block both the Open Directory and Yahoo Directory titles being used, you’ll need to do both of there:

<META NAME="ROBOTS" CONTENT="NOYDIR">
<META NAME="ROBOTS" CONTENT="NOODP">

Maybe. Then again, having two meta robots tags possibly might make a search engine choose one or the other, not both. You could be safe by using a tag for each specific spider:

<META NAME="GOOGLEBOT" CONTENT="NOODP">
<META NAME="SLURP" CONTENT="NOODP">
<META NAME="MSNBOT" CONTENT="NOODP">

But that’s a lot of unnecessary work, likely. Instead, I suspect that you really need to do something like this:

<META NAME="ROBOTS" CONTENT="NOYDIR,NOODP">

That’s consistent with the long-standing standard for the meta robots tag in general, as covered here. And if you’re already using the meta robots tag for other things, such as to block archiving (here are instructions from Google), you probably have to do this:

<META NAME="ROBOTS" CONTENT="NOYDIR,NOODP,NOARCHIVE">

By the way, upper or lower case — it doesn’t matter. I’m also fairly sure that putting spaces after the commas like below works:

<META NAME="ROBOTS" CONTENT="NOYDIR, NOODP, NOARCHIVE">

I’ll do some pinging to get answers to some of the questions above. While it’s great everyone has rolled out support to extend the meta robots tag, they’ve unfortunately not come together to answer clearly some of the questions I’ve raised above.

Postscript: I’ve asked Google, Microsoft and Yahoo the same three questions:

Will there be a problem with two robots tags? Will you go with one or the other (and if so, which one)
Will you recognize a tag like this:
<META NAME="ROBOTS" CONTENT="NOYDIR,NOODP,NOARCHIVE">
Are spaces also OK like this?
<META NAME="ROBOTS" CONTENT="NOYDIR, NOODP, NOARCHIVE">

Postscript, Answers from Yahoo:

Didn’t answer
They do recognize multiple attributes/commands in the same tag, but they didn’t explicitly say if this works if there are no spaces after commas for each attribute (it is probably fine)
Yes, they definitely could handle a tag with multiple attributes and spaces

About The Author: Danny Sullivan is a Founding Editor of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also serves as Chief Content Officer for Third Door Media, which publishes Search Engine Land and produces the SMX: Search Marketing Expo conference series. He has a personal blog called Daggle (and keeps his disclosures page there). He can be found on Facebook, Google + and microblogs on Twitter as @dannysullivan. See more articles by Danny Sullivan

Connect with the author via: Email | Twitter | Google+ | LinkedIn

http://www.mm-agency.com/ Michael Martin

Danny,

Looks like a mistype on the Yahoo Slurp bot (should be META NAME=”Slurp” CONTENT=”NOYDIR”) in this section:

Maybe. Then again, having two meta robots tags possibly might make a search engine choose one or the other, not both. You could be safe by using a tag for each specific spider:

META NAME=”GOOGLEBOT” CONTENT=”NOODP”
META NAME=”SLURP” CONTENT=”NOODP”
META NAME=”MSNBOT” CONTENT=”NOODP”
http://www.luckylester.com Lucky Lester

This new tag that prevents the metadata from being transferred from the Yahoo Directory listings sounds like a good thing and I can’t help but wonder if it will also address the problem of old Directory listings screwing up the organic results?

My point specifically; I have a client site that I have been working with for a while. It is a good site that dates back to 1991. This particular site only lists one page within the Yahoo organic results, the index, and there is no good explanation as to why this is. No spam, no questionable SEO tactics. Applying good research into what might be the cause of this I couldn’t help but notice that Yahoo has multiple listing within their various country specific Directories that are displaying information dating back to pre 1995.

I contacted Yahoo to ask them about this and received a confirmation letter from them stating that it was these old directory listings that were in fact, causing the listing problem. I was told that they would address this issue about 8 months ago and to date there has been no change. (I am still trying to decide if this is a tsk tsk tsk moment for them or does this call for a full blown “A Pox On You Yahoo”)

Anyway, does anybody have any thoughts on whether or not the use of this new tag might help to address this issue? And hey… should any of you Yahoo’s who may read this, care to address this issue once in for all like I was told you would? I am easy to contact, just look for the link sitting comfortably in its condom below.
http://searchengineland.com Danny Sullivan

Michael, not a typo. In that particular example, I was showing you how to use a tag to tell Yahoo to ignore the Open Directory, which you can also do.