Yahoo Search Index Now Supported By Open-Source Hadoop Architecture

Feb 20, 2008 at 9:23am ET by Greg Sterling

As the Yahoo Search Blog explains, open-source Apache Hadoop is now at the center of Yahoo’s search index:

We are now using Hadoop to process the Webmap — the application which produces the index from the billions of pages crawled by Yahoo! Search … Our implementation of a Hadoop-based Webmap is part of a larger strategy of Yahoo! moving toward openness — both in our infrastructure and throughout the network…

There are more technical details here. Hadoop takes over from a proprietary system being used previously. The benefits, among others, are cost savings and scalability.

The irony of this development, however, is that it comes just before Microsoft may take over Yahoo. Microsoft is all about proprietary technology, which is the opposite of what’s going on here. There’s an interview between Jeremy Zawodny and two of the engineers that worked on the project in the video below:

Greg Sterling is a Contributing Editor at Search Engine Land, and writes a personal blog Screenwerk, examining the broader world of media and advertising. He also posts at a Local Mobile Search, which is focused on the mobile Internet.

See more articles by Greg Sterling >

Share, Bookmark & Discuss This Article
More:

Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter

See more stories like this in the Members Library! Check out the Yahoo: Search sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!

Comments are closed.

TOP STORIES

See All »

SEARCH NEWS BRIEFS

See All »

FEATURES & ANALYSIS

See All »

RECENT COMMENTS

Susannah said " I can't wait to try some of these tips this week. What a resource! It's like having a coffee with 21"
dian said " I haven't tried that yet but if it is the way Mazter is saying I think it won't going to do any good"
dian said " It really helps me a lot. The methods step by step explained the important factors easily and all th"

See All »

See How To Articles And More In Our Members Library »

FREE DAILY SEARCH NEWS RECAP!

Stay on top of all the search news with our daily summary, the SearchCap newsletter. View a sample ›

STAY CURRENT THROUGHOUT THE DAY

The Search Engine Land feed keeps you informed as news happens. SEE ALL FEEDS »

SEARCH ENGINE LAND SPONSORS

Advertise With Us »

UPCOMING SEARCH ENGINE LAND CONFERENCES

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.

SMX Web Site » | SMX Difference » | SMX News »

Join us at an upcoming SMX event:

UPCOMING SEARCH ENGINE LAND WEBCASTS

Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include: