Yahoo Search Index Now Supported By Open-Source Hadoop Architecture

Feb 20, 2008 at 9:23am ET by Greg Sterling

As the Yahoo Search Blog explains, open-source Apache Hadoop is now at the center of Yahoo’s search index:

We are now using Hadoop to process the Webmap — the application which produces the index from the billions of pages crawled by Yahoo! Search … Our implementation of a Hadoop-based Webmap is part of a larger strategy of Yahoo! moving toward openness — both in our infrastructure and throughout the network…

There are more technical details here. Hadoop takes over from a proprietary system being used previously. The benefits, among others, are cost savings and scalability.

The irony of this development, however, is that it comes just before Microsoft may take over Yahoo. Microsoft is all about proprietary technology, which is the opposite of what’s going on here. There’s an interview between Jeremy Zawodny and two of the engineers that worked on the project in the video below:

Buzz It

Greg Sterling is a Contributing Editor at Search Engine Land, and writes a personal blog Screenwerk, examining the broader world of media and advertising. He also posts at Internet2Go, which is focused on the mobile Internet.

See more articles by Greg Sterling >

Share, Bookmark & Discuss This Article
More:

Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter

See more stories like this in the Members Library! Check out the Yahoo: Search sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!

Comments are closed.

TOP STORIES

See All »

SEARCH NEWS BRIEFS

See All »

FEATURES & ANALYSIS

See All »

RECENT COMMENTS

nuttakorn said " I think 2010 is about Real-time and personalize search algorithms that Google will pay attention to "
nuttakorn said " I just heard many sources that Google.cn will shut down tomorrow. You can see this source of news fr"
Clifford Bryan said " Interesting concept but I don't see Google being as worried as the article implies. Google has throw"

See All »

See How To Articles And More In Our Members Library »

FREE DAILY SEARCH NEWS RECAP!

SearchCap is a once-per-day newsletter update:

STAY CURRENT THROUGHOUT THE DAY

Our feed & social options update you as news happens.

SEARCH ENGINE LAND SPONSORS

Advertise With Us »

UPCOMING SEARCH ENGINE LAND CONFERENCES

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.

SMX Web Site » | SMX Difference » | SMX News »

Join us at an upcoming SMX event:

UPCOMING SEARCH ENGINE LAND WEBCASTS

Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include: