The Open Directory’s Home Page Goes Missing In Google

The Open Directory’s home page appears to have gone missing from Google’s search results. For example, a search on dmoz (the Open Directory’s nickname) does not return the home page in the search results. Here is a screen capture:

dmoz missing from google

Similarly, searches for open directory or open directory project also don’t list the site at the usual dmoz.org address. Yes, the screenshot shows a page at search.dmoz.org — but normally, the home page would be listed at www.dmoz.org or just dmoz.org (as you can see at Yahoo, Microsoft and Ask, for example).

Google still does have pages from dmoz.org in the index. A search for site:www.dmoz.org clearly returns results but does not appear to return the Open Directory’s home page in those results.

Heck, even a search for www.dmoz.org does not return the Dmoz home page. In addition, the Open Directory’s home page does not come up in Google’s cache.

It is weird that internal pages such as this one come up for normal Google searches but the home page is nowhere to be found. This is even stranger in light of The Great Google Directory Ban Of Sept. 2007. Is the Open Directory somehow being included in an algorithm change that hit smaller directories?

The missing home page is being spotted by many people now (see Bigoakinc.com and Biz Dir Blog), and we have some talk going about it at here at our Sphinn discussion site.

Postscript: Matt Cutts of Google replied to the Sphinn thread explaining that http://www.dmoz.org/ was 301 redirecting (a permanent redirect) back to http://www.dmoz.org/. So there was this loop saying my home page has permanently been changes to my home page. That obviously confused GoogleBot, so after a few days of trying to find the new URL and only being given the old URL, GoogleBot gave up.

Hey all, I dug into this a little bit with the help of a couple crawl folks. It looks like when Googlebot tried to fetch http://www.dmoz.org/, we got a 301 redirect back to http://www.dmoz.org/ . It looks like that self-loop has been going on for several days. We were last able to fetch the root page successfully on Sept. 10th, but from that point on DMOZ was returning these 301-to-itself pages, and after a few days Googlebot gave up on trying to fetch the url. It looks like the rest of the site is fine, so I suspect that if DMOZ gets 301/redirects for their root page sorted out on their webserver, we’ll recrawl and index the page pretty quickly.

So in short, it is an easy fix for the Open Directory Project, but we learned something new. Never 301 redirect a URL to the same URL.

Related Topics: Channel: SEO | Google: SEO | Google: Web Search

Sponsored


About The Author: is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here. For more background information on Barry, see his full bio over here.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://paulpedersen.com/ Paul Pedersen

    Woot! Better late than never.

  • http://www.elixirsystems.com dyland

    Not appearing for a search even for your ‘company’ name (in this case dmoz) can be a sign of a hand edit.

    http://www.seobook.com/how-know-difference-between-automated-penalty-hand-edit

  • http://www.jlh-design.com JLH

    People have been complaining over at Google webmaster’s help group for quite some time about home pages disappearing and no one has bothered to check it out. This semi-popular site doing the same hopefully will encourage them to look into it, but will probably just be a handjob to fix dmoz since it’s gotten so much press lately.

  • http://sethf.com/ Seth Finkelstein

    SPECULATION – this might be a bug in the interaction between Google’s canonicalization routine and redirection issues.

    http:// search.dmoz.org/ does a 301 redirect to http:// dmoz.org/

    http:// dmoz.org/ does a 301 redirect to http:// http://www.dmoz.org/

    I wonder if somehow, it’s deciding that http:// search.dmoz.org/ IS http:// http://www.dmoz.org/ and so then discarding http:// http://www.dmoz.org/

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide