Why MapReduce?

• Traditional Enterprise Systems normally have a centralized

server to store and process data.
• The following illustration depicts a schematic view of a
traditional enterprise system.
• The traditional model is certainly not suitable to process
huge volumes of scalable data and cannot be accommodated
by standard database servers.
• Moreover, the centralized system creates too much of a
bottleneck while processing multiple files simultaneously.
Google solved this bottleneck issue using an algorithm
called MapReduce.
MapReduce divides a task into small parts and assigns them to
many computers. Later, the results are collected at one place
and integrated to form the result dataset.
How MapReduce Works?

 The MapReduce algorithm contains two

important tasks, namely Map and Reduce.
 The Map task takes a set of data and converts it
into another set of data, where individual
elements are broken down into tuples (key-value
pairs).
 The Reduce task takes the output from the Map
as an input and combines those data tuples
(key-value pairs) into a smaller set of tuples.
 The reduce task is always performed after the
map job.
 Let us now take a close look at each of the
phases and try to understand their significance.
• Input Phase − Here we have a Record Reader that translates each
record in an input file and sends the parsed data to the mapper in the
form of key-value pairs.
• Map − Map is a user-defined function, which takes a series of key-
value pairs and processes each one of them to generate zero or more
key-value pairs.
• Intermediate Keys − They key-value pairs generated by the mapper
are known as intermediate keys.
• Combiner − A combiner is a type of local Reducer that groups similar
data from the map phase into identifiable sets. It takes the
intermediate keys from the mapper as input and applies a user-defined
code to aggregate the values in a small scope of one mapper. It is not
a part of the main MapReduce algorithm; it is optional.
• Shuffle and Sort − The Reducer task starts with the Shuffle and Sort
step. It downloads the grouped key-value pairs onto the local machine,
where the Reducer is running. The individual key-value pairs are sorted
by key into a larger data list. The data list groups the equivalent keys
together so that their values can be iterated easily in the Reducer task.
 Reducer − The Reducer takes the grouped key-value paired data
as input and runs a Reducer function on each one of them. Here,
the data can be aggregated, filtered, and combined in a number of
ways, and it requires a wide range of processing. Once the
execution is over, it gives zero or more key-value pairs to the final
step.
 Output Phase − In the output phase, we have an output
formatter that translates the final key-value pairs from the
Reducer function and writes them onto a file using a record writer.
 Tokenize − Tokenizes the tweets into maps of tokens and writes
them as key-value pairs.
 Filter − Filters unwanted words from the maps of tokens and
writes the filtered maps as key-value pairs.
 Count − Generates a token counter per word.
 Aggregate Counters − Prepares an aggregate of similar counter
values into small manageable units.
MapReduce-Example
 Twitter receives around 500 million tweets per
day, which is nearly 3000 tweets per second. The
following illustration shows how Tweeter manages
its tweets with the help of MapReduce.
 Job
IMPALA schedulin
Monitor
g
& Search
Manage

Data
Injestion Nosql(google
Big data)
MapReduce - Algorithm

 The MapReduce algorithm contains two

important tasks, namely Map and Reduce.
 The map task is done by means of Mapper
Class
 The reduce task is done by means of Reducer
Class.
 Mapper class takes the input, tokenizes it,
maps and sorts it. The output of Mapper class
is used as input by Reducer class, which in turn
searches matching pairs and reduces them.
Several ways using which One can Integrate
both R and Hadoop:
 The first layer: It is the hardware layer — it consists of
a cluster of computers systems,
 The second layer: It is the middleware layer of
Hadoop. This layer also takes care of the distributions of
the files flawlessly through using HDFS and the features
of the MapReduce job.
 The third layer: It is the interface layer that provides
the interface for analysis of data. At this level, we can
use an effective tool like Pig which provides a high-level
platform to us for creating MapReduce programs using a
language which we called Pig-Latin. We can also use
Hive which is a data warehouse infrastructure
developed by Apache and built on top of Hadoop. Hive
provides a number of facilities to us for running complex
queries and helps to analyze the data using an SQL-like
language called HiveQL and it also extends support for
implementing MapReduce tasks.
 Besides using Hive and Pig, We can also use
Rhipe or Rhadoop libraries that build an
interface to provide integration between
Hadoop and R and enables users to access
data from the Hadoop file system and enable
to write his own script to implement the Map
and Reduce jobs, or we can also use the
Hadoop- streaming that is a technology
which is used to integrate the Hadoop.
 Using R and Hadoop
There are four different ways of using
Hadoop and R together:

. RHadoop
1

 RHadoop is a collection of three R packages:

rmr, rhdfs and rhbase. rmr package provides
Hadoop MapReduce functionality in R, rhdfs
provides HDFS file management in R and
rhbase provides HBase database
management from within R. Each of these
primary packages can be used to analyze
and manage Hadoop framework data better.
 The rmr package –rmr package provides Hadoop
MapReduce functionality in R. So, the R programmer only has
to do just divide the logic and idea of their application into the
map and reduce phases associates and just submit it with the
rmr methods. After that, The rmr package makes a call to the
Hadoop streaming and the MapReduce API through multiple
job parameters as input directory, output directory, reducer,
mapper, and so on, to perform the R MapReduce job over
Hadoop cluster(most of the components are similar as Hadoop
streaming).
 The rhbase package –Allows R developer to connect Hadoop
HBASE to R using Thrift Server. It also offers functionality like
(read, write, and modify tables stored in HBase from R).
 The rhdfs package –It provides HDFS file management in R,
because data itself stores in Hadoop file system. Functions of
this package are as given as follows. File Manipulations -
( hdfs.delete, hdfs.rm, hdfs.del, hdfs.chown, hdfs.put, hdfs.get
etc), File Read/Write -(hdfs.flush, hdfs.read, hdfs.seek,
hdfs.tell, hdfs.line.reader etc), Directory -hdfs.dircreate,
hdfs.mkdir, Initialization: hdfs.init, hdfs.defaults.
 2. ORCH
 ORCH stands for Oracle R Connector for Hadoop.
 It is a collection of R packages that provide the relevant
interfaces to work with Hive tables, the Apache Hadoop
compute infrastructure, the local R environment, and Oracle
database tables.
 ORCH also provides predictive analytic techniques that can be
applied to data in HDFS files.
 Orch is a collection of R packages that provide the following
features.
 Various attractive Interfaces to work with the data maintained
in Hive tables, able to use the Apache Hadoop based
computing infrastructure, and also provides the local R
environment and Oracle database tables.
 Use a predictive analytic technique, written in R or Java as
Hadoop MapReduce jobs, that can be applied to data stored in
HDFS files
 3. RHIPE

 RHIPE is a R package which provides an API to use Hadoop.

RHIPE stands for R and Hadoop Integrated
Programming Environment, and is essentially RHadoop
with a different API.
 RHIPE: Rhipe is used in R to do an intricate analysis of the
large collection of data sets via Hadoop is an integrated
programming environment tool that is brought by the Divide
and Recombine (D & R) to analyze the huge amount of data.
 Can read, save the complete data that is created using
RHIPE MapReduce. RHIPE is deployed with many features
that help us to effectively interact with HDFS. An individual
can also use various languages like Perl, Java, or Python to
read data sets in RHIPE.
 Rhipe is a Java package, so it acts like a Java bridge between
Hadoop and R. During serialization Rhipe converts the input
data into java types, so that it can be interpreted by Hadoop.
 Hadoop Streaming
 Hadoop streaming is a Hadoop utility for running
the Hadoop MapReduce job with executable
scripts such as Mapper and Reducer.
 The script is available as part of the R package on
CRAN. And its aim is to make R more accessible
to the Hadoop streaming-based applications.
 This is just congruent to the pipe operation in
Linux. With this, the text input file is printed on
stream (stdin), which is provided as an input to
Mapper, and the output (stdout) of Mapper is
provided as an input to the Reducer; finally,
Reducer writes the output to the HDFS directory.
 The main benefit of the Hadoop streaming is to allow the execution of the Java, as
well as non-Java based programmed MapReduce jobs over Hadoop clusters.

 The Hadoop streaming supports various languages like Perl, Python, PHP, R, and
C++, and other programming languages efficiently. Various components of the
Hadoop streaming MapReduce job.