Common Crawl Makes New Web Data Available, Launches Coding Contest
Looking to do research based on data gathered from across the web? That’s one of the purposes of Common Crawl, and the group has just released new data, as well as a contest to encourage use of that data
The 2012 data, which contains 3.8 billion web documents, shows stats such as 63% of top level domains being .com or there being 61 million domains overall.
Common Crawl is also currently running its first-ever Common Crawl Code Contest challenging developers to do something innovative using the data relating to job trends or social impact analysis. Three winners will each get $1,000 in cash, an O’Reilly Data Science Starter Kit, one year of GitHub’s Small Plan and more. Submissions are accepted through August 29.
FYI, I’m on the advisory board for the non-profit group. There’s no compensation for that involvement. I and others just offer free advice to the group.
You can learn more about Common Crawl on its FAQ page, the Get Started page and in the video below:
[youtube width=”560″ height=”315″]https://www.youtube.com/watch?v=ozX4GvUWDm4[/youtube]
Common Crawl’s data from 2011 was recently used by Zyxt Labs to show how much Facebook has spread across the open web. See our Marketing Land article for more on that:
Explore more on
Add Search Engine Land as a preferred source on GoogleEditorial disclosure
Contributing authors are chosen for their expertise and work under the oversight of the editorial staff. Search Engine Land is owned by Semrush. The contributor was not asked to mention Semrush; the opinions expressed are their own.
About the author
