Yahoo Search Index Now Supported By Open-Source Hadoop Architecture


As the Yahoo Search Blog explains, open-source Apache Hadoop is now at the center of Yahoo’s search index:

We are now using Hadoop to process the Webmap — the application which produces the index from the billions of pages crawled by Yahoo! Search … Our implementation of a Hadoop-based Webmap is part of a larger strategy of Yahoo! moving toward openness — both in our infrastructure and throughout the network…

There are more technical details here. Hadoop takes over from a proprietary system being used previously. The benefits, among others, are cost savings and scalability.

The irony of this development, however, is that it comes just before Microsoft may take over Yahoo. Microsoft is all about proprietary technology, which is the opposite of what’s going on here. There’s an interview between Jeremy Zawodny and two of the engineers that worked on the project in the video below:



Greg Sterling is a Contributing Editor at Search Engine Land, and writes a personal blog Screenwerk, examining the broader world of media and advertising. He also posts at Internet2Go, which is focused on the mobile Internet.

See more articles by Greg Sterling >


Share, Bookmark & Discuss This Article
More:


Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter


See more stories like this in the Members Library! Check out the Yahoo: Search sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!

Comments are closed.


RECENT COMMENTS

  • nuttakorn said " I think 2010 is about Real-time and personalize search algorithms that Google will pay attention to "
  • nuttakorn said " I just heard many sources that Google.cn will shut down tomorrow. You can see this source of news fr"
  • Clifford Bryan said " Interesting concept but I don't see Google being as worried as the article implies. Google has throw"

See All »


FREE DAILY SEARCH NEWS RECAP!

SearchCap is a once-per-day newsletter update:

STAY CURRENT THROUGHOUT THE DAY

Our feed & social options update you as news happens.


Advertise With Us »

Search Marketing Expo

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.


SMX Web Site » | SMX Difference » | SMX News »


Join us at an upcoming SMX event:

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:


See more webcast topics »

FOLLOW US SOCIALLY
Upcoming Search Engine Land Conferences

Get Your Search Engine Land
Premium Membership!

Become a premium member today and receive:

  • Express commenting privileges & photo.
  • Exclusive videos & newsletters.
  • Discounts to our SMX conferences.
  • Access to "How To" & Other Archives.

Learn More

Upcoming Search Engine Land Conferences
Add to GoogleAdd to My Yahoo!Add to BloglinesAdd to NetvibesAdd to Windows Live