A Brief Introduction of

Existing Big Data Tools

Bingjing Zhang
Outline
The world map of big data tools
Layered architecture
Big data tools for HPC and supercomputing
MPI
Big data tools on clouds
MapReduce model
Iterative MapReduce model
DAG model
Graph model
Collective model
Machine learning on big data
Query on big data
Stream data processing
The
World of BSP/Collective
Big Data DAG Model MapReduce Model Graph Model
Model
Tools
Hadoop
MPI
HaLoop Giraph
For Twister Hama
Iterations GraphLab
/ Spark GraphX
Learning Harp
Stratosphere
Dryad/ Reef
DryadLI
NQ Pig/PigLatin
Hive
For Tez
Query Drill
Shark
MRQL

For S4 Storm
Streamin Spark
g Samza
Streaming
Layered

BioKe
stratio

Workf

(Tools

BioKe
NA:
stratio
Orche

Swift,
Workf

(Tools
Oozie

Keple

Galax
NA:
Pegas
ODE,
Orche
Architecture

Swift,

eBPE
Airav
Oozie

Keple

Galax
Taver

Activ
Pegas
ODE,

OOD

eBPE
Airav

Tride
Taver

Activ
OOD

Tride

pler,
pler,
&

and
low
nn &

and
low

na,
ata

us,

na,
ata

nt,
us,

nt,

L,
L,
TT

r,r,

yy
))
,,
(Upper) Cross Cutting
Capabilities
Machine Learning Data Analytics Libraries:

Statistics, Bioinformatics Imagery Linear Algebra

Mahout , MLlib , MLbase
R, Bioconductor (NA) ImageJ (NA) Scalapack, PetSc (NA)

Monitoring:
Monitoring: Ambari,
CompLearn (NA)

Message
Message Protocols:
High Level (Integrated) Systems for Data Processing

Distributed
Distributed Coordination:
Hive Hcatalog Pig Shark MRQL Impala (NA) Swazall
(SQL on Interfaces (Procedural (SQL on (SQL on Hadoop, Cloudera (Log Files

Protocols: Thrift,
Ambari, Ganglia,
Security
Hadoop) Language) Spark, NA) Hama, Spark) (SQL on Hbase) Google NA)

Security &

Coordination: ZooKeeper,
Parallel Horizontally Scalable Data Processing

JGroups

JGroups
NA Non Apache projects Pegasus

Ganglia, Nagios,
Hadoop Spark NA:Twister Tez Hama S4 Samza Giraph on Hadoop

& Privacy

Thrift, Protobuf
(Map (Iterative Stratosphere (DAG) Storm
(BSP) Yahoo LinkedIn ~Pregel

Privacy
Green layers are Reduce) MR) Iterative MR
(NA)

Stream Graph

Nagios, Inca
Batch

Protobuf (NA)
Apache/Commercial Cloud (light)

ZooKeeper,
ABDS Inter-process Communication HPC Inter-process Communication
to HPC (darker) integration layers
Inca (NA)
Hadoop, Spark Communications MPI (NA)

(NA)
(NA) & Reductions Harp Collectives (NA)
Pub/Sub Messaging Netty (NA)/ZeroMQ (NA)/ActiveMQ/Qpid/Kafka

The figure of layered architecture is from Prof.

Geoffrey Fox
 Cross Cutting In memory distributed databases/caches: GORA (general object from NoSQL), Memcached
(NA), Redis(NA) (key value), Hazelcast (NA), Ehcache (NA);
Layered Capabilities
ORM Object Relational Mapping: Hibernate(NA), OpenJPA and JDBC Standard
Extraction Tools SQL NoSQL: Column Solandra
Architecture UIMA
(Entities)
Tika MySQL Phoenix
SciDB
(NA)
Arrays,
(SQL on
HBase Accumulo
Cassandra
(Data on
(Solr+
Cassandra)
(Data on

(Lower)
(Content) (NA) R,Python (DHT) +Document
(Watson) HBase) HDFS) HDFS)

NoSQL: Document NoSQL: Key Value (all NA)

MongoDB
CouchDB Lucene Berkeley Azure Dynamo Riak Voldemort

Distributed
Distributed Coordination:
Monitoring:
Monitoring: Ambari,
(NA)
Solr DB Table Amazon ~Dynamo ~Dynamo

Message
Message Protocols:
NoSQL: General Graph NoSQL: TripleStore RDF SparkQL File
Management

Coordination: ZooKeeper,
Protocols: Thrift,
Neo4J Yarcdata

Ambari, Ganglia,
Security
Sesame AllegroGraph RYA RDF on

Security &
Java Gnu Commercial Jena
NA Non Apache projects (NA) (NA)
(NA) Commercial Accumulo iRODS(NA)

Ganglia, Nagios,
Green layers are

& Privacy

Thrift, Protobuf
Data Transport BitTorrent, HTTP, FTP, SSH Globus Online (GridFTP)

Privacy
ZooKeeper, JGroups
Apache/Commercial Cloud (light)

Nagios, Inca
HPC Cluster Resource Management

Protobuf (NA)
to HPC (darker) integration layers ABDS Cluster Resource Management
Mesos, Yarn, Helix, Llama(Cloudera) Condor, Moab, Slurm, Torque(NA) ..

Inca (NA)

(NA)
JGroups
(NA) ABDS File Systems
HDFS, Swift, Ceph User Level
FUSE(NA) HPC FileGluster,
SystemsLustre,
(NA) GPFS, GFFS

Object Stores POSIX Interface Distributed, Parallel, Federated

Interoperability Layer Whirr / JClouds OCCI CDMI (NA)
DevOps/Cloud Deployment Puppet/Chef/Boto/CloudMesh(NA)
IaaS System Manager Open Source Commercial Clouds
Bare
OpenStack, OpenNebula, Eucalyptus, CloudStack, vCloud, Amazon, Azure, Google
Metal
The figure of layered architecture is from Prof.
Geoffrey Fox
Big Data Tools for HPC and
Supercomputing
MPI(Message Passing Interface, 1992)
Provide standardized function interfaces for communication between
parallel processes.

Collective communication operations

Broadcast, Scatter, Gather, Reduce, Allgather, Allreduce, Reduce-scatter.

Popular implementations
MPICH (2001)
OpenMPI (2004)
http://www.open-mpi.org/
MapReduce Model
Google MapReduce (2004)
Jeffrey Dean et al. MapReduce: Simplified Data Processing on Large Clusters.
OSDI 2004.

Apache Hadoop (2005)

http://hadoop.apache.org/
http://developer.yahoo.com/hadoop/tutorial/

Apache Hadoop 2.0 (2012)

Vinod Kumar Vavilapalli et al. Apache Hadoop YARN: Yet Another Resource
Negotiator, SOCC 2013.
Separation between resource management and computation model.
Key Features of MapReduce Model
Designed for clouds
Large clusters of commodity machines

Designed for big data

Support from local disks based distributed file system (GFS / HDFS)
Disk based intermediate data transfer in Shuffling

MapReduce programming model

Computation pattern: Map tasks and Reduce tasks
Data abstraction: KeyValue pairs
Google MapReduce
Mapper: split, read, emit User Reducer: repartition,
intermediate KeyValue Program emits final output
pairs
(1) fork (1) (1) fork
(2) forkMaster (2)
assign assig
map n
Worker reduc (6) write Output
e Worker
Split 0 (3) read File 0
(4) local
Split 1 Worker write
Split 2 Output
Worker
File 1
Worker (5) remote
read
Intermediate files
Input files Map phase Reduce phase Output files
(on local disks)
Iterative MapReduce Model
Twister
(2010)
Jaliya Ekanayake et al. Twister: A Runtime for Iterative MapReduce. HPDC workshop
2010.
http://www.iterativemapreduce.org /
Simple collectives: broadcasting and aggregation.

HaLoop (2010)
Yingyi Bu et al. HaLoop: Efficient Iterative Data Processing on Large clusters. VLDB
2010.
http://code.google.com/p/haloop /
Programming model
Loop-Aware Task Scheduling
Caching and indexing for Loop-Invariant Data on local disk
Twister Programming Model
Main programs process Worker Nodes
space
configureMaps() Local
configureReduce( Disk
)
while(condition){ Cacheable map/reduce
tasks
runMapReduce(.
May scatter/broadcast Map
..)
<Key,Value> pairs directly ()
Iteration May merge data in shuffling Reduce
s Combine() ()
Communications/data transfers
operation via the pub-sub broker network &
direct TCP
updateCondition( Main program may contain many
)} //end while
MapReduce invocations or
close() iterative MapReduce invocations
DAG (Directed Acyclic Graph) Model
Dryad and DryadLINQ (2007)
Michael Isard et al. Dryad: Distributed Data-Parallel Programs
from Sequential Building Blocks, EuroSys, 2007.
http://
research.microsoft.com/en-us/collaboration/tools/dryad.aspx
Model Composition
Apache Spark (2010)
Matei Zaharia et al. Spark: Cluster Computing with Working Sets,.
HotCloud 2010.
Matei Zaharia et al. Resilient Distributed Datasets: A Fault-Tolerant
Abstraction for In-Memory Cluster Computing. NSDI 2012.
http://spark.apache.org/
Resilient Distributed Dataset (RDD)
RDD operations
MapReduce-like parallel operations
DAG of execution stages and pipelined transformations
Simple collectives: broadcasting and aggregation
Graph Processing with BSP model
Pregel (2010)
Grzegorz Malewicz et al. Pregel: A System for Large-Scale Graph Processing.
SIGMOD 2010.

Apache Hama (2010)

https://hama.apache.org/

Apache Giraph (2012)

https://giraph.apache.org/
Scaling Apache Giraph to a trillion edges
https://www.facebook.com/notes/facebook-engineering/scaling-apache-giraph-to-a-tri
llion-edges/10151617006153920
Pregel & Apache Giraph
Vote to Active
Computation Model
halt
Superstep as iteration
Vertex state machine: 3 6 2 1 Superste
Active and Inactive, vote to halt p0
Message passing between vertices
Combiners 6 6 2 6 Superste
Aggregators p1
Topology mutation
Master/worker model 6 6 6 6 Superste
p2
Graph partition: hashing
Fault tolerance: checkpointing and 6 6 6 6 Superste
confined recovery p3
Maximum Value Example
Giraph Page Rank Code Example
public class PageRankComputation
extends BasicComputation<IntWritable, FloatWritable, NullWritable, FloatWritable> {
/** Number of supersteps */
public static final String SUPERSTEP_COUNT = "giraph.pageRank.superstepCount";

@Override
public void compute(Vertex<IntWritable, FloatWritable, NullWritable> vertex, Iterable<FloatWritable> messages)
throws IOException {
if (getSuperstep() >= 1) {
float sum = 0;
for (FloatWritable message : messages) {
sum += message.get();
}
vertex.getValue().set((0.15f / getTotalNumVertices()) + 0.85f * sum);
}
if (getSuperstep() < getConf().getInt(SUPERSTEP_COUNT, 0)) {
sendMessageToAllEdges(vertex,
new FloatWritable(vertex.getValue().get() / vertex.getNumEdges()));
} else {
vertex.voteToHalt();
}
}
}
GraphLab (2010)

Yucheng Low et al. GraphLab: A New Parallel Framework for Machine

Learning. UAI 2010.
Yucheng Low, et al. Distributed GraphLab: A Framework for Machine Learning
and Data Mining in the Cloud. PVLDB 2012.
http://graphlab.org/projects/index.html
http://graphlab.org/resources/publications.html

Data graph
Update functions and the scope
Sync operation (similar to aggregation in Pregel)
Data Graph
Vertex-cut v.s. Edge-cut

PowerGraph (2012)
Joseph E. Gonzalez et al. PowerGraph:
Distributed Graph-Parallel Computation
on Natural Graphs. OSDI 2012.
Gather, apply, Scatter (GAS) model

GraphX (2013)
Reynold Xin et al. GraphX: A Resilient
Edge-cut (Giraph
Distributed Graph System on Spark. model)
GRADES (SIGMOD workshop) 2013.
https
://amplab.cs.berkeley.edu/publication/gr
aphx-grades/ Vertex-cut (GAS model)
To reduce communication
overhead.
Option 1
Algorithmic message reduction
Fixed point-to-point communication pattern

Option 2
Collective communication optimization
Not considered by previous BSP model but well developed in MPI
Initial attempts in Twister and Spark on clouds
Mosharaf Chowdhury et al. Managing Data Transfers in Computer Clusters with
Orchestra. SIGCOMM 2011.
Bingjing Zhang, Judy Qiu. High Performance Clustering of Social Images in a Map-
Collective Programming Model. SOCC Poster 2013.
Collective Model
Harp (2013)
https://github.com/jessezbj/harp-project
Hadoop Plugin (on Hadoop 1.2.1 and Hadoop 2.2.0)
Hierarchical data abstraction on arrays, key-values and graphs for
easy programming expressiveness.
Collective communication model to support various communication
operations on the data abstractions.
Caching with buffer management for memory allocation required
from computation and communication
BSP style parallelism
Fault tolerance with check-pointing
Harp Design

Parallelism Model Architecture

MapReduce Map-Collective Model

MapReduce Map-Collective
Model Application Applications Applications
M M M M
Harp
M M M M Framework
Shuffle Collective
MapReduce V2
Communication
R R Resource YARN
Manager
Hierarchical Data Abstraction
and Collective Communication
Broadcast, Allgather, Allreduce, Regroup-
(combine/reduce),
Array Table Edge Message-to-Vertex,
Message Edge-to-Vertex
Vertex KeyValue
Table Table Table Table Table
<Array Type>

Array Partition
Edge Message Vertex KeyValue
Partition < Array Type
Partition Partition Partition Partition
>
Broadcast, Send

Long Int Double Byte Vertices, Edges,

Key-Values
Array Array Array Array Messages

Basic Types Array Struct Object

Broadcast, Send,
Gather
Commutable
Harp Bcast Code Example
protected void mapCollective(KeyValReader reader, Context context)
throws IOException, InterruptedException {

ArrTable<DoubleArray, DoubleArrPlus> table =

new ArrTable<DoubleArray, DoubleArrPlus>(0, DoubleArray.class, DoubleArrPlus.class);

if (this.isMaster()) {
String cFile = conf.get(KMeansConstants.CFILE);
Map<Integer, DoubleArray> cenDataMap = createCenDataMap(cParSize, rest, numCenPartitions,

vectorSize, this.getResourcePool());
loadCentroids(cenDataMap, vectorSize, cFile, conf);
addPartitionMapToTable(cenDataMap, table);
}

arrTableBcast(table);
}
Broadcasting with Twister vs. MPI Twister vs. MPJ
(Broadcasting 0.5~2GB data) (Broadcasting 0.5~2GB data)
25 40
35

Topology-
20
30

15 25

Awareness
20
10 15
10
5
5
0 0
1 25 50 75 100 125 150 1 25 50 75 100 125 150
Number of Nodes Number of Nodes
Twis ter Bcas t 500MB MPI Bcas t 500MB
Twis ter 0.5GB MPJ 0.5GB
Twis ter Bcas t 1GB MPI Bcas t 1GB
Twis ter Bcas t 2GB MPI Bcas t 2GB Twis ter 1GB MPJ 1GB

Twister vs. Spark (Broadcasting Twister Chain with/without

80 0.5GB data) 100 topology-awareness
90
70
80
60 70
50 60
40 50
40
30
30
20 20
10 10
0 0
1 25 50 75 100 125 150 1 25 50 75 100 125 150
Number of Nodes Number of Nodes

1 receiver 0.5GB 0.5GB W/O TA

#receivers = #nodes 1GB 1GB W/O TA
#receivers = #cores (#nodes *8)

Tested on IU Polar Grid with 1 Gbps Ethernet

connection
K-Means Clustering Performance on
Madrid Cluster (8 nodes)
K-Means Clustering Harp v.s. Hadoop on Madrid
1600

1400

1200

1000

Execution Time (s) 800

600

400

200

0
100m 500 10m 5k 1m 50k

Problem Size

Hadoop 24 cores Harp 24 cores Hadoop 48 cores Harp 48 cores Hadoop 96 cores Harp 96 cores
K-means
Clustering Parallel
Efficiency

Shantenu Jha et al. A Tale of Two

Data-Intensive Paradigms:
Applications, Abstractions, and
Architectures. 2014.
WDA-MDS Performance on
Big Red II
WDA-MDS
Yang Ruan, Geoffrey Fox. A Robust and Scalable Solution for
Interpolative Multidimensional Scaling with Weighting. IEEE e-
Dcience 2013.
Big Red II
http://kb.iu.edu/data/bcqt.html
Allgather
Bucket algorithm
Allreduce
Bidirectional exchange algorithm
Execution Time of 100k Problem
Execution Time of 100k Problem
3000

2500

2000

Execution Time (Seconds) 1500

1000

500

0
0 20 40 60 80 100 120 140

Number of Nodes (8, 16, 32, 64, 128 nodes, 32 cores per node)
Parallel Efficiency
Based On 8 Nodes and 256 Cores
Parallel Efficiency (Based On 8Nodes and 256 Cores)
1.2

0.8

0.6

0.4

0.2

0
0 20 40 60 80 100 120 140

Number of Nodes (8, 16, 32, 64, 128 nodes)

4096 partitions (32 cores per node)

Scale Problem Size (100k, 200k,
300k)
Scaling Problem Size on 128 nodes with 4096 cores
3500

3000 2877.76

2500

2000
1643.08
Execution Time (Seconds)
1500

1000

500 368.39

0
100000 200000 300000

Problem Size
Machine Learning on Big Data
Mahout on Hadoop
https://mahout.apache.org/

MLlib on Spark
http://spark.apache.org/mllib/

GraphLab Toolkits
http://graphlab.org/projects/toolkits.html
GraphLab Computer Vision Toolkit
Query on Big Data
Query with procedural language

Google Sawzall (2003)

Rob Pike et al. Interpreting the Data: Parallel Analysis with Sawzall.
Special Issue on Grids and Worldwide Computing Programming Models
and Infrastructure 2003.

Apache Pig (2006)

Christopher Olston et al. Pig Latin: A Not-So-Foreign Language for Data
Processing. SIGMOD 2008.
https://pig.apache.org/
SQL-like Query
Apache Hive (2007)
Facebook Data Infrastructure Team. Hive - A Warehousing Solution Over a Map-
Reduce Framework. VLDB 2009.
https://hive.apache.org/
On top of Apache Hadoop
Shark (2012)
Reynold Xin et al. Shark: SQL and Rich Analytics at Scale. Technical Report.
UCB/EECS 2012.
http://shark.cs.berkeley.edu/
On top of Apache Spark
Apache MRQL (2013)
http://mrql.incubator.apache.org /
On top of Apache Hadoop, Apache Hama, and Apache Spark
Other Tools for Query
Apache Tez (2013)
http://tez.incubator.apache.org/
To build complex DAG of tasks for Apache Pig and Apache Hive
On top of YARN

Dremel (2010) Apache Drill (2012)

Sergey Melnik et al. Dremel: Interactive Analysis of Web-Scale
Datasets. VLDB 2010.
http://incubator.apache.org/drill/index.html
System for interactive query
Stream Data Processing
Apache S4 (2011)
http://incubator.apache.org/s4/

Apache Storm (2011)

http://storm.incubator.apache.org/

Spark Streaming (2012)

https://spark.incubator.apache.org/streaming/

Apache Samza (2013)

http://samza.incubator.apache.org/
REEF
Retainable Evaluator Execution Framework
http://www.reef-project.org/
Provides system authors with a centralized (pluggable) control
flow
Embeds a user-defined system controller called the Job Driver
Event driven control
Package a variety of data-processing libraries (e.g., high-
bandwidth shuffle, relational operators, low-latency group
communication, etc.) in a reusable form.
To cover different models such as MapReduce, query, graph
processing and stream data processing
Thank You!
Questions?