Mining Public Datasets

Mining Public Datasets
using Apache Zeppelin (incubating),

Apache Spark and Juju
by Alexander Bezzubov
NFLabs for AppacheCon ’16 NA

Alexander Bezzubov
Software Engineer at NFLabs, Seoul,

South Korea
Co-organizer of SeoulTech Society
Committer and PPMC member of

Apache Zeppelin (Incubating)
github.com/bzz
@seoul_engineer
Graduated Maths at St.Petersburg State
University, Russia
PUBLIC DATASETS: Number, Size & Growth
Web Crawls
Structured data (RDF, micro-formats, tables)
Hackers News\Reddit\Twitter\StackOverflow\Wikipedia
Reviews (movies, restaurants, beer, wine)
Emails (Enroll, ASF public ML archives)
Census Data (US, UK, UN, Japan, etc)
Transportation (Taxi, Flights, Bicycles)
Climate
Genome
Web Crawls
order of Tbs
Genome
Web Crawls
order of Tbs
Genome
AWS Public Datasets https://aws.amazon.com/public-data-sets/
Yahoo Webscope https://webscope.sandbox.yahoo.com/
Stanford Network Analyser Project http://snap.stanford.edu/data/
Physics Research http://opendata.cern.ch

Web Crawls
order of Tbs
Genome
AWS Public Datasets https://aws.amazon.com/public-data-sets/
Yahoo Webscope https://webscope.sandbox.yahoo.com/
Stanford Network Analyser Project http://snap.stanford.edu/data/
Physics Research http://opendata.cern.ch order of Pbs

PUBLIC DATA = OPPORTUNITY
I. Tools
II. Data
TOOL TO PURSUIT THE OPPORTUNITY:
Overview Big Data eco-system
… …
Overview Big Data eco-system
Todays choice Zeppelin, Spark, Juju
Apache Spark
Scala, Python, R
Apache Zeppelin
Modern Web GUI, plays nicely with Spark, Flink,
Elasticsearch, etc.
Warcbase
Spark library for saved crawl data (WARC)
Juju
Scales, integration with Spark, Zeppelin, AWS, GCE
APACHE ZEPPELIN: Overview

Zeppelin: Brief history
http://zeppelin.incubator.apache.org
12.2012 Commercial App using AMP Lab Shark 0.5

10.2013 Prototype Hive/Shark
08.2013 NFLabs Internal project Hive/Shark
12.2014 Enters ASF Incubation
01.2016 3 major releases
05.2016 Graduation vote passed
Interactive Visualization
APACHE SPARK
http://spark.apache.org
From Berkeley AMP Labs, since 2010
Joined Apache since 2014
1000+ contributors
REPL + Java, Scala, Python, R APIs

JUJU
https://jujucharms.com/
Service modelling at scale
Deployment\configuration automation
+ Integration with Spark, Zeppelin, Ganglia, etc
+ AWS, GCE, Azure, LXC, etc
JUJU
http://bigdata.juju.solutions/getstarted
$ apt-get install juju-core juju-quickstart

# or
$ brew install juju juju-quickstart
$ juju generate-config
#LXC, AWS, GCE, Azure, VMWare, OpenStack
$ juju bootstrap
$ juju quickstart apache-hadoop-spark-zeppelin
$ juju expose spark zeppelin
$ juju add-unit -n4 slave
JUJU
http://bigdata.juju.solutions/getstarted
7 node cluster designed to scale out

APPROACH: local, small cluster, big cluster
1 core Prototype
Your laptop
10s PC Estimate the cost

AWS spot instances
1000 instances Scale out Deployment automation

I. Tools
II. Data
DATA: GitHub http://githubarchive.org
• 300Gb compressed
• Collaboration google and github engineers
• Events on PR, repo, issues, comments, etc in JSON

http://www.commitlogsfromlastnight.com/
http://sideeﬀect.kr/popularconvention/
https://www.gitlive.net/
http://zoom.it/kCsU
DATA PRODUCT: Get notified when
project goes Open Source
DATA PRODUCT: Exploration
DATA PRODUCT: Sketch
We are going to build a Notebook that

sends you a digest email:
DATA PRODUCT: pieces (flow-chart)
We are going to build a Notebook that:

• Downloads the latest data from GitHub Archive
• Read & explore the dataset
• Imports, filters the PublicEvent
• Join logs w/ more data from Github API calls
• Shows HTML template, to visualise the list
• Sends email notifications
• Does all above automatically, once a day

DATA PRODUCT: Full impl
I. Tools
II. Data
DATA: Common Crawl
https://commoncrawl.org
Nonprofit, by Factual
On AWS S3 in WARC, WAT, formats
since 2013, monthly: ~150Tb compressed, 2+bln ulrs
URL Index by Ilya Kreymer of @webrecorder_io
http://index.commoncrawl.org/
https://about.commonsearch.org
DATA: CommonCrawl - Data Product
Measuring the impact of Google Analytics
Objective: estimate % of pages/domains that use Google

Analytics/Facebook
Existing research from 2013

Measuring the impact of Google Analytics

Copy to HDFS vs read from S3
Verify using grep
hadoop jar hadoop-examples.jar grep /grep-data/ \
/grep-output/ '[Bb]ig [Dd]ata is ([a-zA-Z]{5,})' 
Verify using grep

Feb 2016 Crawl: 

- 48Tb compressed
- 100 segments (dir on S3)
- 30,000 files, ~1Gb each
AWS optimisations:
- pick spot instance prices
- pick instance type (net throughput)
- user Juju instead of EMR (2x $$ savings!)
Spark optimisations:
- IO-bound, so increase spark.executor.cores
spark.executor.memory
Zeppelin Viewer
Community service for sharing example notebooks

http://zeppelinhub.com/viewer
TAKEAWAY
There are plenty of free tools out there
To crunch the data for fun and profit
They are easy (not simple) to learn and generic enough

Questions?
Alexander Bezzubov
@seoul_engineer
github.com/bzz
Thank you
Alexander Bezzubov
NFLabs, Seoul (we are hiring!)

Mining Public Datasets

Uploaded by

Copyright:

Available Formats

Mining Public Datasets

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mining Public Datasets

Uploaded by

Copyright:

Available Formats

Mining Public Datasets

using Apache Zeppelin (incubating),

NFLabs for AppacheCon ’16 NA

Software Engineer at NFLabs, Seoul,

Co-organizer of SeoulTech Society

Committer and PPMC member of

Physics Research http://opendata.cern.ch

Physics Research http://opendata.cern.ch order of Pbs

APACHE ZEPPELIN: Overview

12.2012 Commercial App using AMP Lab Shark 0.5

From Berkeley AMP Labs, since 2010

Joined Apache since 2014

REPL + Java, Scala, Python, R APIs

Service modelling at scale

+ Integration with Spark, Zeppelin, Ganglia, etc

+ AWS, GCE, Azure, LXC, etc

$ apt-get install juju-core juju-quickstart

7 node cluster designed to scale out

10s PC Estimate the cost

1000 instances Scale out Deployment automation

• Collaboration google and github engineers

• Events on PR, repo, issues, comments, etc in JSON

We are going to build a Notebook that

We are going to build a Notebook that:

• Read & explore the dataset

• Imports, filters the PublicEvent

• Join logs w/ more data from Github API calls

• Shows HTML template, to visualise the list

• Sends email notifications

• Does all above automatically, once a day

On AWS S3 in WARC, WAT, formats

since 2013, monthly: ~150Tb compressed, 2+bln ulrs

URL Index by Ilya Kreymer of @webrecorder_io

Measuring the impact of Google Analytics

Objective: estimate % of pages/domains that use Google

Existing research from 2013

Measuring the impact of Google Analytics

Verify using grep

Feb 2016 Crawl:

Community service for sharing example notebooks

There are plenty of free tools out there

To crunch the data for fun and profit

They are easy (not simple) to learn and generic enough

You might also like

Feb 2016 Crawl: 