Common Crawl: Difference between revisions

{{Short description|Nonprofit web crawling and archive organization}}

| name = Common Crawl

| company_type = [[501(c)(3)]] non-profit

| traded_as =

| foundation = 2007

| dissolved = =

| location = [[San Francisco, California]]; [[Los Angeles, California]], United States

| incorporated = =

| founder = [[Gil Elbaz]]

| chairman =

| president =
| CEO =

| industry products =

| products services =

| services assets =

| assets equity =

| equity owner =

| parent divisions =

| alexa website_type =

| website_type registration =

| registration num_users =

| launch_date current_status =

| current_status screenshot =

| screenshot caption =

| caption footnotes =

'''Common Crawl''' is a [[nonprofit organization|nonprofit]] [[501(c) organization#501.28c.29.283.29|501(c)(3)]] organization that [[web crawler|crawls]] the web and freely provides its archives and datasets to the public.<ref name=latimes>{{cite news |title=Tech entrepreneur Gil Elbaz made it big in L.A.|author=Rosanna Xia|work=Los Angeles Times|date=February 5, 2012|access-date=July 31, 2014|url=httphttps://articleswww.latimes.com/2012business/la-xpm-2012-feb/-05/business/-la-fi-himi-elbaz-20120205-story.html}}</ref><ref name=pressheretv>{{cite news |title=Gil Elbaz and Common Crawl|work=NBC News|date=April 4, 2013|access-date=July 31, 2014|url=http://www.pressheretv.com/gil-elbaz-and-common-crawl/}}</ref> Common Crawl's [[Web archiving|web archive]] consists of [[petabyte|petabytes]] of data collected since 20112008.<ref name="ready">{{cite web |url=https://commoncrawl.org/the-data/get-started/ |title=So you're ready to get started|publisher=Common Crawl|access-date=2018-06-029 June 2023}}</ref> It completes crawls generally every month.<ref name=theverge>{{cite news |title=Winter 2013 Crawl Data Now Available|author=Lisa Green|date=January 8, 2014|access-date=June 2, 2018|url=https://commoncrawl.org/2014/01/winter-2013-crawl-data-now-available/}}</ref>

Common Crawl was founded by [[Gil Elbaz]].<ref name=twist>{{cite news |title=Startups - Gil Elbaz and Nova Spivack of Common Crawl - TWiST #222|publisher=This Week In Startups|date=January 10, 2012}}</ref> Advisors to the non-profit include [[Peter Norvig]] and [[Joi Ito]].<ref name=technologyreview>{{cite news |title=A Free Database of the Entire Web May Spawn the Next Google|author=Tom Simonite|publisher=MIT Technology Review|date=January 23, 2013|access-date=July 31, 2014|url=http://www.technologyreview.com/news/509931/a-free-database-of-the-entire-web-may-spawn-the-next-google/|archive-date=June 26, 2014|archive-url=https://web.archive.org/web/20140626114525/http://www.technologyreview.com/news/509931/a-free-database-of-the-entire-web-may-spawn-the-next-google/|url-status=dead}}</ref> The organization's crawlers respect [[nofollow]] and [[Robot exclusion standard|robots.txt]] policies. Open source code for processing Common Crawl's data set is publicly available.

[[Amazon Web Services]] began hosting Common Crawl's archive through its Public Data Sets program in 2012.<ref name=semanticweb_1>{{cite news |title=Common Crawl Toto Add New Data Inin Amazon Web Services Bucket |author=Jennifer Zaino |publisher=Semantic Web |date=March 13, 2012 |access-date=July 31, 2014 |url=http://semanticweb.com/common-crawl-to-add-new-data-in-amazon-web-services-bucket_b27341 |archive-url=https://web.archive.org/web/20140701235708/http://semanticweb.com/common-crawl-to-add-new-data-in-amazon-web-services-bucket_b27341 |archive-date=July 1, 2014 |url-status=dead}}</ref>

The organization began releasing [[metadata]] files and the text output of the crawlers alongside [[ARCHeritrix#Arc (file format)files|.arc]] files in July of that year2012.<ref name=semanticweb_2>{{cite news |author=Jennifer Zaino |date=July 16, 2012 |title=Common Crawl Corpus Update Makes Web Crawl Data More Efficient, Approachable Forfor Users Toto Explore|author=Jennifer Zaino|publisher=Semantic Web|date=July 16, 2012|access-date=July 31, 2014|url=http://semanticweb.com/common-crawl-corpus-update-makes-web-crawl-data-more-efficient-approachable-for-users-to-explore_b30771 |archive-url=https://web.archive.org/web/20140812101154/http://semanticweb.com/common-crawl-corpus-update-makes-web-crawl-data-more-efficient-approachable-for-users-to-explore_b30771 |archive-date=August 12, 2014 |url-status=dead |publisher=Semantic Web |access-date=July 31, 2014}}</ref> Common Crawl's archives had only included .arc files previously.<ref name=semanticweb_2 />

In December 2012, [[blekko]] donated to Common Crawl search engine [[metadata]] blekko had gathered from crawls it conducted from February to October 2012.<ref name=semanticweb_3>{{cite news |author=Jennifer Zaino |date=December 18, 2012 |title=Blekko Data Donation Is As Big Benefit Toto Common Crawl|author=Jennifer Zaino|publisher=Semantic Web|date=December 18, 2012|access-date=July 31, 2014|url=http://semanticweb.com/blekko-data-donation-is-a-big-benefit-to-common-crawl_b34177 |archive-url=https://web.archive.org/web/20140812101151/http://semanticweb.com/blekko-data-donation-is-a-big-benefit-to-common-crawl_b34177 |archive-date=August 12, 2014 |url-status=dead |publisher=Semantic Web |access-date=July 31, 2014}}</ref> The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive [[search engine optimization|SEO]]."<ref name=semanticweb_3 />

In 2013, Common Crawl began using the [[Apache Software Foundation|Apache Software Foundation]]'s]] [[Nutch]] webcrawler instead of a custom crawler.<ref name=ccnutch>{{cite web |title=Common Crawl's Move to Nutch|author=Jordan Mendelson|publisher=Common Crawl|date=February 20, 2014 |access-datetitle=JulyCommon 31,Crawl's Move to Nutch 2014|url=http://commoncrawl.org/common-crawl-move-to-nutch/ |publisher=Common Crawl |access-date=July 31, 2014}}</ref> Common Crawl switched from using .arc files to [[Web ARChive |.warc]] files with its November 2013 crawl.<ref name=ccnov2013>{{cite web |title=New Crawl Data Available!|author=Jordan Mendelson|publisher=Common Crawl|date=November 27, 2013 |access-datetitle=JulyNew 31,Crawl Data Available! 2014|url=http://commoncrawl.org/new-crawl-data-available/ |publisher=Common Crawl |access-date=July 31, 2014}}</ref>

A filtered version of Common Crawl was used to train OpenAI's [[GPT-3]] language model, announced in 2020.<ref>{{Cite arXiv |last1=Brown |first1=Tom |last2=Mann |first2=Benjamin |last3=Ryder |first3=Nick |last4=Subbiah |first4=Melanie |last5=Kaplan |first5=Jared |last6=Dhariwal |first6=Prafulla |last7=Neelakantan |first7=Arvind |last8=Shyam |first8=Pranav |last9=Sastry |first9=Girish |last10=Askell |first10=Amanda |last11=Agarwal |first11=Sandhini |date=2020-06-01|page=14 |title=Language Models areAre Few-Shot Learners |classpage=cs.CL14 |eprint=2005.14165 |class=cs.CL |quote="the majority of our data is derived from raw Common Crawl with only quality-based filtering."}}</ref> One challenge of using Common Crawl data is that despite the vast amount of documented web data, individual pieces of crawled websites could be better documented. This can create challenges when trying to diagnose problems in projects that use the Common Crawl data. A solution proposed by [[Timnit Gebru]], et al., in 2020 to an industry-wide documentation shortfall is that every dataset should be accompanied with a datasheet that documents its motivation, composition, collection process, and recommended uses.<ref>{{cite arXiv| first1 = Timnit | last1 = Gebru | first2 = Jamie | last2 = Morgenstern | first3 = Briana | last3 = Vecchione | first4 = Jennifer | last4 = Wortman Vaughan | first5 = Hanna | last5 = Wallach| first6 = Hal | last6 = Daumé III | first7 = Kate | last7 = Crawford | title = Datasheets for Datasets| date = March 19, 2020 | class = cs.DB | eprint = 1803.09010 }}</ref>

==HistoryTimeline of Common Crawl data==

The following data have been collected from the official Common Crawl Blog.<ref>{{Cite web|url=httphttps://commoncrawl.org/connect/blog/|title=Blog – Common Crawl}}</ref>

! Crawl date !! Size in [[Tebibyte|TiB]] !! Billions of pages !! Comments

|—220

| JanuaryWinter 20142013 || 148 || 2.3 || CrawlsCrawl performedconducted monthlyfrom December 4 through December 22, 2013

| NovemberSummer 2013 || 102? || 2? || DataCrawl inconducted Warcfrom fileMay format2013 through June 2013. First WARC crawl

| July 2012 || —? || —? || DataCrawl inconducted from January 2012 through June 2012. ArcFinal fileARC formatcrawl

| January 20122009-2010 || —? || —? || PublicCrawl conducted Datafrom SetJuly of2009 Amazonthrough WebSeptember Services2010

*[https://commoncrawl.org/connect/blog/ Common Crawl Blog]