SlideShare a Scribd company logo
Martin Ahchiev
Content is available under a Creative Commons 3.0 License unless otherwise noted.
2
05.12.2015
Apache Spark
• What is Big Data?
• Internet of Things
• What is Apache Spark?
• History of Apache Spark
• Why Spark?
• Spark Execution Flow
• Spark Context
• Resilient Distributed Dataset (RDD)
• RDD Examples
• MapReduce Algorithm
• MapReduce Example: Word Count
• Let’s try some examples
3
05.12.2015
What is Big Data?
Apache Spark
• “Big data” is similar to “small
data”, but bigger in size – is not
incorrect
• But having data bigger it
requires different approaches
• Big Data is a set of technologies and methods for handling
large volumes of data at rapid speeds and of various
formats.
4
05.12.2015
Big Data and the Internet of Things
Apache Spark
• Connected Intelligence
• Every day, we create 2.5 quintillion bytes of data — so much that
90% of the data in the world today has been created in the last two
years alone. IBM, “Bringing big data to the enterprise”
5
05.12.2015
In Next 60 seconds…
Apache Spark
6
05.12.2015
Big Data – Trends and Opportunities
Apache Spark
”Welcome to the Internet of Customers. Behind every app, every device,
and every connection, is a customer. Billions of them. And each and every
one is speeding toward the future.” Salesforce.com
7
05.12.2015
What is Apache Spark?
Apache Spark
• Emerging big data framework
• Open source framework for fast distributed in-memory
data processing and data analytics
• Extension/alternative to the MapReduce model
• Currently an Apache high-priority “top-level” project
8
05.12.2015
What is Apache Spark?
Apache Spark
• Written in Scala
 Functional programming language that runs in a JVM
• Key Concepts
 Avoid the data bottleneck by distributing data when it is stored
 Bring the processing to the data
 Data is stored in memory
9
05.12.2015
History of Apache Spark
Apache Spark
• Started in UC Berkeley AMPLab as a research
project by Matei Zaharia, 2009
 AMP = Algorithms Machines People
 AMPLab is integrating Algorithms, Machines, and People to
make sense of Big Data
• Spark become open source, March 2010
• Spark donated to Apache Software Foundation,
June 2013
• Spark becomes a top-level Apache project,
February 2014
10
05.12.2015
Why Spark?
Apache Spark
Speed
• Run programs up to 100x faster than Hadoop MapReduce in
memory, or 10x faster on disk.
• Last year, Spark took over Hadoop by completing the 100 TB
Daytona GraySort contest 3x faster on one tenth the number of
machines and it also became the fastest open source engine for
sorting a petabyte.
11
05.12.2015
Why Spark?
Apache Spark
Ease of Use
• Write applications quickly in Java, Scala, Python, R.
• Spark offers over 80 high-level operators that make it
easy to build parallel apps. And you can use it
interactively from the Scala, Python and R shells.
12
05.12.2015
Why Spark?
Apache Spark
Generality
• Combine SQL, streaming, and complex analytics
• Spark powers a stack of libraries including SQL and
DataFrames, MLlib for machine learning, GraphX, and Spark
Streaming. You can combine these libraries seamlessly in the
same application
13
05.12.2015
Why Spark?
Apache Spark
Runs Everywhere
• Spark runs on Hadoop, Mesos, standalone, or in the cloud. It
can access diverse data sources including HDFS, Cassandra,
HBase, and S3.
• You can run Spark using its standalone cluster mode, on EC2,
on Hadoop YARN, or on Apache Mesos. Access data in
HDFS, Cassandra, HBase, Hive, Tachyon, and any Hadoop
data source.
14
05.12.2015
Execution Flow
Apache Spark
• Cluster Manager An external service to manage resources on the cluster
(standalone manager, YARN, Apache Mesos)
• Worker Node : Node that run the application program in cluster
• Executor
1. Process launched on a worker node, that runs the Tasks
2. Keep data in memory or disk storage
• Task : A unit of work that will be sent to executor
• Job
1. Consists multiple tasks
2. Created based on a Action
• Stage : Each Job is divided into smaller set of tasks called Stages that is sequential
and depend on each other
• SparkContext : represents the connection to a Spark cluster, and can be used to
create RDDs, accumulators and broadcast variables on that cluster.
• Driver Program
The process to start the execution (main() function)
15
05.12.2015
Spark Context
Apache Spark
• Every Spark application requires a Spark Context
 The main entry point to the Spark API
• Spark Shell Provides a preconfigured Spark Context call sc
16
05.12.2015
Resilient Distributed Dataset (RDD)
Apache Spark
• RDD is a basic Abstraction in Spark
• Distributed collection of objects
• RDD(Resilient Distributed Dataset)
1. Resilient – if data in memory is lost, it can be recreated
2. Distributed – stored in memory across the cluster
3. Dataset – initial data can come from a file or created
programmatically
17
05.12.2015
Example: A File-base RDD
Apache Spark
18
05.12.2015
Example: A File-base RDD
Apache Spark
19
05.12.2015
RDD Operations
Apache Spark
• Two types of RDD operations
 Actions – return values
 count
 take(n)
 Transformations – define new RDDs based
on the current one
 filter
 map
 reduce
20
05.12.2015
Example map and filter Transformations
Apache Spark
21
05.12.2015
RDDs
Apache Spark
• RDDs can hold any type of element
 Primitive types: integers, chars, Boolean, string, etc
 Sequence type: lists, arrays, tuples, dicts, etc.
 Scala/Java Object (if serializable)
• Some types of RDDs have additional
functionality
 Double RDDs – RDDs consisting of numeric data
 Pair RDDs – RDDs consisting of Key-Value pairs
22
05.12.2015
Pair RDDs
Apache Spark
• Pair RDDs are a special form of RDD
 Each element must be a key-value pair
 Keys and values can be any type
• Why?
 Use with Map-Reduce algorithms
 Many additional functions are available for common data
processing needs – e.g. sorting, joining, grouping,
counting, etc
23
05.12.2015
MapReduce
Apache Spark
• MapReduce is a common programming model
1. Two phases
 Map – process each element in a data set
 Reduce – aggregate or consolidate the data
2. Easily applicable to distributed processing of large data sets
• Hadoop MapReduce is the major implementation
1. Limited
 Each job has one Map phase, one Reduce phase in each
 Job output saved to files
• Spark implements MapReduce with much greater flexibility
1. Map and Reduce functions can be interspersed
2. Results stored in memory
 Operations can be
chained easily
Spark execution flow
Hadoop execution flow
24
05.12.2015
MapReduce
Apache Spark
25
05.12.2015
MapReduce (Contd.)
Apache Spark
26
05.12.2015
MapReduce (Contd.)
Apache Spark
27
05.12.2015
MapReduce (Contd.)
Apache Spark
28
05.12.2015
MapReduce (Contd.)
Apache Spark
29
05.12.2015
MapReduce (Contd.)
Apache Spark
30
05.12.2015
MapReduce Example: Word Count
Apache Spark
31
05.12.2015
MapReduce Example: Word Count
Apache Spark
32
05.12.2015
MapReduce Example: Word Count
Apache Spark
33
05.12.2015
MapReduce Example: Word Count
Apache Spark
34
05.12.2015
MapReduce Example: Word Count
Apache Spark
35
05.12.2015
MapReduce Example: Word Count
Apache Spark
36
05.12.2015
MapReduce Example: Word Count
Apache Spark
37
05.12.2015
ReduceByKey
Apache Spark
• ReduceByKey functions must be
 Binary: combines values from two keys
 Commutative: x+y = y+x
 Associative: (x+y)+z = x+(y+z)
38
05.12.2015
MapReduce Example: Word Count
Apache Spark
39
05.12.2015
MapReduce Example: Word Count
Apache Spark
Let’s try some examples
40
05.12.2015
Apache Spark
martin.ahchiev@musala.com

More Related Content

PDF
An Overview of Apache Spark
Yasoda Jayaweera
 
PDF
Spark Core
Todd McGrath
 
PDF
Apache Spark Briefing
Thomas W. Dinsmore
 
PDF
Introduction to Apache Spark
Juan Pedro Moreno
 
PDF
Apache Spark 1.6 with Zeppelin - Transformations and Actions on RDDs
Timothy Spann
 
PDF
Introduction to Apache Spark
datamantra
 
PDF
Introduction to apache spark
Aakashdata
 
PDF
Big Telco - Yousun Jeong
Spark Summit
 
An Overview of Apache Spark
Yasoda Jayaweera
 
Spark Core
Todd McGrath
 
Apache Spark Briefing
Thomas W. Dinsmore
 
Introduction to Apache Spark
Juan Pedro Moreno
 
Apache Spark 1.6 with Zeppelin - Transformations and Actions on RDDs
Timothy Spann
 
Introduction to Apache Spark
datamantra
 
Introduction to apache spark
Aakashdata
 
Big Telco - Yousun Jeong
Spark Summit
 

What's hot (20)

PDF
Introduction to apache spark
UserReport
 
PDF
Transitioning Compute Models: Hadoop MapReduce to Spark
Slim Baltagi
 
PDF
Geospatial Analytics at Scale with Deep Learning and Apache Spark with Tim hu...
Databricks
 
PPTX
Introduction to Apache Spark
Rahul Jain
 
PDF
Apache Spark for RDBMS Practitioners: How I Learned to Stop Worrying and Lov...
Databricks
 
PPTX
Spark - The Ultimate Scala Collections by Martin Odersky
Spark Summit
 
PPTX
Real time Analytics with Apache Kafka and Apache Spark
Rahul Jain
 
PPTX
Lambda architecture with Spark
Vincent GALOPIN
 
PDF
Hadoop and Spark
Shravan (Sean) Pabba
 
PDF
Flink in Zalando's world of Microservices
ZalandoHayley
 
PPTX
Spark Advanced Analytics NJ Data Science Meetup - Princeton University
Alex Zeltov
 
PDF
Big Data visualization with Apache Spark and Zeppelin
prajods
 
PPTX
Apache Spark Fundamentals
Zahra Eskandari
 
PPTX
Data Science with Spark & Zeppelin
Vinay Shukla
 
PPTX
Using Visualization to Succeed with Big Data
Pactera_US
 
PPTX
Future of data visualization
hadoopsphere
 
PDF
Introduction to Apache Spark
Samy Dindane
 
PPTX
Apache spark - History and market overview
Martin Zapletal
 
PPTX
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...
Alex Zeltov
 
PPTX
Spark and Spark Streaming
宇 傅
 
Introduction to apache spark
UserReport
 
Transitioning Compute Models: Hadoop MapReduce to Spark
Slim Baltagi
 
Geospatial Analytics at Scale with Deep Learning and Apache Spark with Tim hu...
Databricks
 
Introduction to Apache Spark
Rahul Jain
 
Apache Spark for RDBMS Practitioners: How I Learned to Stop Worrying and Lov...
Databricks
 
Spark - The Ultimate Scala Collections by Martin Odersky
Spark Summit
 
Real time Analytics with Apache Kafka and Apache Spark
Rahul Jain
 
Lambda architecture with Spark
Vincent GALOPIN
 
Hadoop and Spark
Shravan (Sean) Pabba
 
Flink in Zalando's world of Microservices
ZalandoHayley
 
Spark Advanced Analytics NJ Data Science Meetup - Princeton University
Alex Zeltov
 
Big Data visualization with Apache Spark and Zeppelin
prajods
 
Apache Spark Fundamentals
Zahra Eskandari
 
Data Science with Spark & Zeppelin
Vinay Shukla
 
Using Visualization to Succeed with Big Data
Pactera_US
 
Future of data visualization
hadoopsphere
 
Introduction to Apache Spark
Samy Dindane
 
Apache spark - History and market overview
Martin Zapletal
 
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...
Alex Zeltov
 
Spark and Spark Streaming
宇 傅
 
Ad

Similar to Big Data Processing with Apache Spark 2014 (20)

PPT
Big_data_analytics_NoSql_Module-4_Session
RUHULAMINHAZARIKA
 
PPTX
Pyspark presentationsfspfsjfspfjsfpsjfspfjsfpsjfsfsf
sasuke20y4sh
 
PPTX
Unit II Real Time Data Processing tools.pptx
Rahul Borate
 
PPTX
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
Simplilearn
 
PPTX
APACHE SPARK.pptx
DeepaThirumurugan
 
PPTX
In Memory Analytics with Apache Spark
Venkata Naga Ravi
 
PDF
Spark Driven Big Data Analytics
inoshg
 
PPTX
Spark core
Prashant Gupta
 
PPTX
Getting Started with Apache Spark (Scala)
Knoldus Inc.
 
PDF
Started with-apache-spark
Happiest Minds Technologies
 
PPTX
An Introduction to Apache Spark
Dona Mary Philip
 
PPTX
Apache spark
Prashant Pranay
 
PPTX
Apache Spark on HDinsight Training
Synergetics Learning and Cloud Consulting
 
PPTX
4Introduction+to+Spark.pptx sdfsdfsdfsdfsdf
yafora8192
 
PPTX
Apache Spark Introduction @ University College London
Vitthal Gogate
 
PPTX
Apache Spark Core
Girish Khanzode
 
PPTX
Apache Spark II (SparkSQL)
Datio Big Data
 
PPTX
Spark with anjbnn hfkkjn hbkjbu h jhbk.pptx
nreddyjanga
 
PDF
Spark forspringdevs springone_final
sdeeg
 
PDF
Apache Spark Introduction
sudhakara st
 
Big_data_analytics_NoSql_Module-4_Session
RUHULAMINHAZARIKA
 
Pyspark presentationsfspfsjfspfjsfpsjfspfjsfpsjfsfsf
sasuke20y4sh
 
Unit II Real Time Data Processing tools.pptx
Rahul Borate
 
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
Simplilearn
 
APACHE SPARK.pptx
DeepaThirumurugan
 
In Memory Analytics with Apache Spark
Venkata Naga Ravi
 
Spark Driven Big Data Analytics
inoshg
 
Spark core
Prashant Gupta
 
Getting Started with Apache Spark (Scala)
Knoldus Inc.
 
Started with-apache-spark
Happiest Minds Technologies
 
An Introduction to Apache Spark
Dona Mary Philip
 
Apache spark
Prashant Pranay
 
Apache Spark on HDinsight Training
Synergetics Learning and Cloud Consulting
 
4Introduction+to+Spark.pptx sdfsdfsdfsdfsdf
yafora8192
 
Apache Spark Introduction @ University College London
Vitthal Gogate
 
Apache Spark Core
Girish Khanzode
 
Apache Spark II (SparkSQL)
Datio Big Data
 
Spark with anjbnn hfkkjn hbkjbu h jhbk.pptx
nreddyjanga
 
Spark forspringdevs springone_final
sdeeg
 
Apache Spark Introduction
sudhakara st
 
Ad

Recently uploaded (20)

PPTX
Probability systematic sampling methods.pptx
PrakashRajput19
 
PDF
Research about a FoodFolio app for personalized dietary tracking and health o...
AustinLiamAndres
 
PPTX
Introduction to Biostatistics Presentation.pptx
AtemJoshua
 
PDF
Linux OS guide to know, operate. Linux Filesystem, command, users and system
Kiran Maharjan
 
PPTX
Data-Driven Machine Learning for Rail Infrastructure Health Monitoring
Sione Palu
 
PPTX
Introduction to Data Analytics and Data Science
KavithaCIT
 
PDF
Company Profile 2023 PT. ZEKON INDONESIA.pdf
hendranofriadi26
 
PDF
Key_Statistical_Techniques_in_Analytics_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PDF
Key_Statistical_Techniques_in_Analytics_by_CA_Suvidha_Chaplot (1).pdf
CA Suvidha Chaplot
 
PPTX
Measurement of Afordability for Water Supply and Sanitation in Bangladesh .pptx
akmibrahimbd
 
PPTX
INFO8116 - Week 10 - Slides.pptx big data architecture
guddipatel10
 
PPTX
Economic Sector Performance Recovery.pptx
yulisbaso2020
 
PPTX
artificial intelligence deeplearning-200712115616.pptx
revathi148366
 
PDF
The_Future_of_Data_Analytics_by_CA_Suvidha_Chaplot_UPDATED.pdf
CA Suvidha Chaplot
 
PPTX
Pipeline Automatic Leak Detection for Water Distribution Systems
Sione Palu
 
PDF
Data_Cleaning_Infographic_Series_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PPTX
Employee Salary Presentation.l based on data science collection of data
barridevakumari2004
 
PPTX
Data Security Breach: Immediate Action Plan
varmabhuvan266
 
PPTX
Trading Procedures (1).pptxcffcdddxxddsss
garv794
 
PPTX
Presentation1.pptxvhhh. H ycycyyccycycvvv
ItratBatool16
 
Probability systematic sampling methods.pptx
PrakashRajput19
 
Research about a FoodFolio app for personalized dietary tracking and health o...
AustinLiamAndres
 
Introduction to Biostatistics Presentation.pptx
AtemJoshua
 
Linux OS guide to know, operate. Linux Filesystem, command, users and system
Kiran Maharjan
 
Data-Driven Machine Learning for Rail Infrastructure Health Monitoring
Sione Palu
 
Introduction to Data Analytics and Data Science
KavithaCIT
 
Company Profile 2023 PT. ZEKON INDONESIA.pdf
hendranofriadi26
 
Key_Statistical_Techniques_in_Analytics_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
Key_Statistical_Techniques_in_Analytics_by_CA_Suvidha_Chaplot (1).pdf
CA Suvidha Chaplot
 
Measurement of Afordability for Water Supply and Sanitation in Bangladesh .pptx
akmibrahimbd
 
INFO8116 - Week 10 - Slides.pptx big data architecture
guddipatel10
 
Economic Sector Performance Recovery.pptx
yulisbaso2020
 
artificial intelligence deeplearning-200712115616.pptx
revathi148366
 
The_Future_of_Data_Analytics_by_CA_Suvidha_Chaplot_UPDATED.pdf
CA Suvidha Chaplot
 
Pipeline Automatic Leak Detection for Water Distribution Systems
Sione Palu
 
Data_Cleaning_Infographic_Series_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
Employee Salary Presentation.l based on data science collection of data
barridevakumari2004
 
Data Security Breach: Immediate Action Plan
varmabhuvan266
 
Trading Procedures (1).pptxcffcdddxxddsss
garv794
 
Presentation1.pptxvhhh. H ycycyyccycycvvv
ItratBatool16
 

Big Data Processing with Apache Spark 2014

  • 1. Martin Ahchiev Content is available under a Creative Commons 3.0 License unless otherwise noted.
  • 2. 2 05.12.2015 Apache Spark • What is Big Data? • Internet of Things • What is Apache Spark? • History of Apache Spark • Why Spark? • Spark Execution Flow • Spark Context • Resilient Distributed Dataset (RDD) • RDD Examples • MapReduce Algorithm • MapReduce Example: Word Count • Let’s try some examples
  • 3. 3 05.12.2015 What is Big Data? Apache Spark • “Big data” is similar to “small data”, but bigger in size – is not incorrect • But having data bigger it requires different approaches • Big Data is a set of technologies and methods for handling large volumes of data at rapid speeds and of various formats.
  • 4. 4 05.12.2015 Big Data and the Internet of Things Apache Spark • Connected Intelligence • Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. IBM, “Bringing big data to the enterprise”
  • 5. 5 05.12.2015 In Next 60 seconds… Apache Spark
  • 6. 6 05.12.2015 Big Data – Trends and Opportunities Apache Spark ”Welcome to the Internet of Customers. Behind every app, every device, and every connection, is a customer. Billions of them. And each and every one is speeding toward the future.” Salesforce.com
  • 7. 7 05.12.2015 What is Apache Spark? Apache Spark • Emerging big data framework • Open source framework for fast distributed in-memory data processing and data analytics • Extension/alternative to the MapReduce model • Currently an Apache high-priority “top-level” project
  • 8. 8 05.12.2015 What is Apache Spark? Apache Spark • Written in Scala  Functional programming language that runs in a JVM • Key Concepts  Avoid the data bottleneck by distributing data when it is stored  Bring the processing to the data  Data is stored in memory
  • 9. 9 05.12.2015 History of Apache Spark Apache Spark • Started in UC Berkeley AMPLab as a research project by Matei Zaharia, 2009  AMP = Algorithms Machines People  AMPLab is integrating Algorithms, Machines, and People to make sense of Big Data • Spark become open source, March 2010 • Spark donated to Apache Software Foundation, June 2013 • Spark becomes a top-level Apache project, February 2014
  • 10. 10 05.12.2015 Why Spark? Apache Spark Speed • Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. • Last year, Spark took over Hadoop by completing the 100 TB Daytona GraySort contest 3x faster on one tenth the number of machines and it also became the fastest open source engine for sorting a petabyte.
  • 11. 11 05.12.2015 Why Spark? Apache Spark Ease of Use • Write applications quickly in Java, Scala, Python, R. • Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python and R shells.
  • 12. 12 05.12.2015 Why Spark? Apache Spark Generality • Combine SQL, streaming, and complex analytics • Spark powers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application
  • 13. 13 05.12.2015 Why Spark? Apache Spark Runs Everywhere • Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3. • You can run Spark using its standalone cluster mode, on EC2, on Hadoop YARN, or on Apache Mesos. Access data in HDFS, Cassandra, HBase, Hive, Tachyon, and any Hadoop data source.
  • 14. 14 05.12.2015 Execution Flow Apache Spark • Cluster Manager An external service to manage resources on the cluster (standalone manager, YARN, Apache Mesos) • Worker Node : Node that run the application program in cluster • Executor 1. Process launched on a worker node, that runs the Tasks 2. Keep data in memory or disk storage • Task : A unit of work that will be sent to executor • Job 1. Consists multiple tasks 2. Created based on a Action • Stage : Each Job is divided into smaller set of tasks called Stages that is sequential and depend on each other • SparkContext : represents the connection to a Spark cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster. • Driver Program The process to start the execution (main() function)
  • 15. 15 05.12.2015 Spark Context Apache Spark • Every Spark application requires a Spark Context  The main entry point to the Spark API • Spark Shell Provides a preconfigured Spark Context call sc
  • 16. 16 05.12.2015 Resilient Distributed Dataset (RDD) Apache Spark • RDD is a basic Abstraction in Spark • Distributed collection of objects • RDD(Resilient Distributed Dataset) 1. Resilient – if data in memory is lost, it can be recreated 2. Distributed – stored in memory across the cluster 3. Dataset – initial data can come from a file or created programmatically
  • 19. 19 05.12.2015 RDD Operations Apache Spark • Two types of RDD operations  Actions – return values  count  take(n)  Transformations – define new RDDs based on the current one  filter  map  reduce
  • 20. 20 05.12.2015 Example map and filter Transformations Apache Spark
  • 21. 21 05.12.2015 RDDs Apache Spark • RDDs can hold any type of element  Primitive types: integers, chars, Boolean, string, etc  Sequence type: lists, arrays, tuples, dicts, etc.  Scala/Java Object (if serializable) • Some types of RDDs have additional functionality  Double RDDs – RDDs consisting of numeric data  Pair RDDs – RDDs consisting of Key-Value pairs
  • 22. 22 05.12.2015 Pair RDDs Apache Spark • Pair RDDs are a special form of RDD  Each element must be a key-value pair  Keys and values can be any type • Why?  Use with Map-Reduce algorithms  Many additional functions are available for common data processing needs – e.g. sorting, joining, grouping, counting, etc
  • 23. 23 05.12.2015 MapReduce Apache Spark • MapReduce is a common programming model 1. Two phases  Map – process each element in a data set  Reduce – aggregate or consolidate the data 2. Easily applicable to distributed processing of large data sets • Hadoop MapReduce is the major implementation 1. Limited  Each job has one Map phase, one Reduce phase in each  Job output saved to files • Spark implements MapReduce with much greater flexibility 1. Map and Reduce functions can be interspersed 2. Results stored in memory  Operations can be chained easily Spark execution flow Hadoop execution flow
  • 37. 37 05.12.2015 ReduceByKey Apache Spark • ReduceByKey functions must be  Binary: combines values from two keys  Commutative: x+y = y+x  Associative: (x+y)+z = x+(y+z)
  • 39. 39 05.12.2015 MapReduce Example: Word Count Apache Spark Let’s try some examples