Buy Search Indexes Off The Shelf From 80legs

The startup 80legs launched in September of last year to make web crawling affordable and accessible to anyone that wanted to subscribe to the service. The company makes crawling the web or customized slices of it available to even small companies. More flexible than something like the Google Custom Search Engine, pricing ranges from free to $300 per month for users or companies with “heavy crawling needs.”

After watching how its customers were using the service the company has now come out what it calls “Crawl Packages,” pre-configured crawls of various sorts. Think of them as “vertical” or niche indexes. These include social networks and shopping sites:

Picture 68

Here’s the early list of these pre-configured crawls:

The data being crawled are only those that are publicly available, CEO Shion Deysarkar explained. He said that many of 80legs’ clients were doing interesting analysis or building functionality on top of the crawls being generated by 80legs.

There are a wide range of applications for the service, including for consumer-facing sites and B2B services, such as social media monitoring. Deysarkar also gave the example of product sites being crawled to monitor pricing data. Another example: a startup that sought to add rental or for-sale property listings could use 80legs to collect that information from the top-10 real estate sites.

One could crawl and analyze all the public Facebook and LinkedIn profiles, and so on. The service offers a cost-effective way to build an index or monitor certain types of content or activity online.

Related Topics: Channel: Consumer | Search Engines: Custom Search Engines


About The Author: is a Contributing Editor at Search Engine Land. He writes a personal blog Screenwerk, about SoLoMo issues and connecting the dots between online and offline. He also posts at Internet2Go, which is focused on the mobile Internet. Follow him @gsterling.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide