Learning SEO From Building A Web Crawler
Apr 29, 2009 at 9:10am ET by Barry Schwartz
There is no doubt that you can learn a tremendous amount of information on search engine optimization (SEO) by reading sites like this or ones in our blogroll, but there is always a lot to be learned from getting your hands dirty. Now, you can get your hands dirty by experimenting and trying SEO techniques out on sites and you can also learn an incredible amount by trying to reverse engineer a web crawler by building your own.
In fact, Google Webmaster Analyst, JohnMu, tweeted this morning stating that fact. He said, “Want to learn about indexing/crawling? Don’t read – code a spider.”
That is exactly what SEOmoz did, they built a crawler and index of web pages to better learn about the internet, plus share that data with the industry. Linkscape was introduced in October 2008 and has grown to 44 billion web pages and 474 billion links.
Rand Fishkin of SEOmoz has posted the lessons learned from building an index of the web. So, maybe, in this case, reading about someone else’s experiences and findings in building such a crawler can help you.
Share, Bookmark & Discuss This Article
More:
Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter
See more stories like this in the Members Library! Check out the SEM Tools, SEO: General sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!
TOP STORIES
SEARCH NEWS BRIEFS
FEATURES & ANALYSIS
RECENT COMMENTS
- poiriem said " Excellent article, and yes it is a real problem - especially for agencies - managing budgets and KPI"
- AdvertiseSpace said " Thank you for sharing, Google maps are very helpful in almost all categories of business. I am glad "
- AdvertiseSpace said " I am a music person and I hope Google will make this all happen. This will serve as a big threat to "
FREE DAILY SEARCH NEWS RECAP!
Stay on top of all the search news with our daily summary, the SearchCap newsletter. View a sample ›
STAY CURRENT THROUGHOUT THE DAY
Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.
SMX Web Site » | SMX Difference » | SMX News »
Join us at an upcoming SMX event:
- SMX West - March 2-4, 2010
- SMX Munich - March 23-24, 2010
- SMX Toronto - April 8-9, 2010
- SMX Sydney - April 22-23, 2010
- SMX Advanced London - May 17-18, 2010
- SMX Advanced Seattle - June 8-9, 2010
- SMX East - October 4-6, 2010
- SMX Stockholm - October, 2010
Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:
- PPC Advertising & Geo-Targeting - November 12, 2009
- Making Online Marketing Relevant and Responsive: Tactics and A Tool for Success - November 17, 2009
- Ask the Search Engines: War Stories from the World Tour - December 1, 2009
TRACK US SOCIALLY
- Wall Street Journal Recommended Reading
- Ad Age Power 150 Blog
- Technorati Top 100 Blog
- Techmeme Leaderboard Source
- Best Search Engine Community/Forum
Featured sites from our Blogroll
Get Your Search Engine Land
Premium Membership!
Become a premium member today and receive:
- Express commenting privileges & photo.
- Exclusive videos & newsletters.
- Discounts to our SMX conferences.
- Access to "How To" & Other Archives.
© 2009 Third Door Media, Inc. All rights reserved. Google, Yahoo & Microsoft logos trademark respective companies and used with permission.
