BIG DATA ANALYTICS Sri Sairam College of Engineering

Department of CSE
BIG DATA ANALYTICS

Module 1

Chapter 1: Hadoop Distributed File System

Hadoop Distributed File System Design Features

The Hadoop Distributing File System (HDFS) was designed for Big Data processing. The design
assumes a large file write-one/read-many model that enables other optimizations and relaxes many of
the concurrency and coherence overhead requirements of a true parallel file system.

The design of HDFS is based on the design of the Google File System (GFS). HDFS is
designed for data streaming where large amounts of data are read from disk in bulk. The HDFS block
size is typically 64MB or 128MB. Thus, this approach is entirely unsuitable for standard POSIX file
system use. The large block and file sizes make it more efficient to reread data from HDFS than to
try to cache the data.

A principal design aspect of Hadoop MapReduce is the emphasis on moving the computation
to the data rather than moving the data to the computation. This distinction is reflected in how
Hadoop clusters are implemented. In other high-performance systems, a parallel file system will
exist on hardware separate from the compute hardware. Data is then moved to and from the
computer components via high-speed interfaces to the parallel file system array.

Finally, Hadoop clusters assume node (and even rack) failure will occur at some point. To
deal with this situation, HDFS has a redundant design that can tolerate system failure and still
provide the data needed by the compute part of the program.

The following points summarize the important aspects of HDFS:

 The write-once/read-many design is intended to facilitate streaming reads.
 Files may be appended, but random seeks are not permitted. There is no caching of data.
BIG DATA ANALYTICS Sri Sairam College of Engineering

 Converged data storage and processing happen on the same server nodes.
 "Moving computation is cheaper than moving data."
 A reliable file system maintains multiple copies of data across the cluster.
 A specialized file system is used,w hich is not designed for general use.
Module-1 BDA-15CS82

HDFS Components
The design of HDFS is based on two types of nodes: a NameNode and multiple DataNodes. A single
NameNode manages all the metadata node to store and retrieve the actual data from the DataNodes.
No data is actually stored on the NameNode, however .For a minimal Hadoop installation, there
needs to be a single NameNode daemon and a single DataNode daemon running machine on at least
one machine.
The design is a master/slave architecture in which the master (NameNode) manages the file
system namespace and regulates access to files by clients. File system namespace operations such as
opening, closing, and renaming files and directories are all managed by the NameNode. The
NameNode also determines the mapping of blocks to DataNodes and handles DataNode failures.
The slaves (DataNodes) are responsible for serving read and w rite requests from the file
system to the clients. The NameNode manages block creation, deletion, and replication.
An example of the client/NameNode/DataNode interaction is provided in Figure 1.1. When a client
writes data, it first communicates with the NameNode and requests to create a file.

Figure 1.1: Various system roles in an HDFS deployment

The NameNode determines how many blocks are needed and provides the client with the DataNodes
that will store the data. As part of the storage process, the data blocks are replicated after they are
written to the assigned node. Depending on how many nodes are in the cluster, the Namenode will
attempt to write replicas of the data blocks on nodes that are in other separate racks. If he is only one

Department of CSE Page 2

Module-1 BDA-15CS82

rack, then the replicated blocks are written to other servers in the same rack. After the DataNode
acknowledges that the file block replication is complete, the client closes the file and informs the
NameNode that the operation is complete. Note that the NameNode does not write any data directly
to the DataNodes. It does, however, give the client a limited amount of time to complete the
operation. If it does not complete in the time period, the operation is canceled.
Reading data happens in a similar fashion. The client requests a file from the NameNode,
which returns the best DataNodes from which to read the data. The client then accesses the data
directly from the DataNodes.
Thus, once the metadata has been delivered to the client, the NameNode steps back and lets
the conversation between the client and the DataNodes proceed. While data transfer is progressing,
the NameNode also monitors the DataNodes by listening for heartbeats sent from DataNodes. The
lack of a heartbeat signal indicates a potential node failure. In such a case, the NameNode will route
around the failed DataNode and begin re-replicating the now-missing blocks. Because the file system
is redundant, DataNodes can be taken offline (decommissioned) for maintenance by informing the
NameNode of the DataNodes to exclude from the HDFS pool.
The mappings between data blocks and the physical DataNodes are not kept in persistent
storage on the NameNode. For performance reasons, the NameNode stores all metadata in memory.
Upon startup, each DataNode provides a block report (which it keeps in persistent storage) to the
NameNode. The block reports are sent every 10 heartbeats.
In almost all Hadoop deployments, there is a SecondaryNameNode While not explicitly
required by a NameNode, it is highly recommended. The term "Secondary-NameNode " (now called
CheckPointNode) is somewhat misleading. It is not an active failover node and cannot replace the
primary NameNode in case of its failure.
The purpose of the secondary NameNode is to perform periodic checkpoint that evaluate the
status of the NameNode. Recall that the NameNode keeps all system metadata memory for fast
access. It also has two disk files that track changes to the metadata:
 An image of the file system state when the NameNode was started. This file begins with
fsimage_* and is used only at startup by the NameNode.
 As series of modification done to the file system after starting the NameNode. These files
begin with edit_* and reflect the changes made after the fsimage_* file was read.
The location of these files is set by the dts.namenode.dir property in the hdfs-Site.starting xml file.

Department of CSE Page 3

Module-1 BDA-15CS82

HDFS Block Replication

As mentioned, when HDFS writes a file., it is replicated across the cluster. The amount based on the
value of dfs. replication in the hdfs-site.xml file. This default value can be overruled with the hdfs
dfs-setrep command. For Hadoop clusters containing more than eight Data Nodes, the replication
value is usually set to 3.
The HDFS default block size is often 64MB. The block size is 4KB or 8KB. The HDFS
default block size not the minimum block size, however. If a 20KB file is written to HDFS, it will
create a block that is approximately 20 in size. If a file of size 80MB is Written to HDFS, a 64MB
block and a 16MB block will be created.
Figure 1.2 provides an example of how a file is broken into blocks and replicated across the
cluster. In this case, a replication factor of 3 ensures that any one DataNode can fail and the
replicated blocks will be available on other nodes-and then subsequently re-replicated on other
DataNodes.

Figure 1.2: HDFS block replication example

HDFS Safe Mode

When the NameNode starts, it enters a read-only safe mode where blocks cannot be replicated or
deleted. Safe Mode enables the NameNode to perform two important processes:

1. The previous file system state is reconstructed by loading the fsimage file into memory and
replaying the edit log.
2. The mapping between blocks and data nodes is created by waiting for enough of the DataNodes to
register so that atleast one copy of the data is available. Not all DataNodes are required to register

Department of CSE Page 4

Module-1 BDA-15CS82

before HDFS exits from Safe Mode. The registration process may continue for some time.
HDFS may also enter Safe Mode for maintenance using the hdfs dfsadmin-safemode command or
when there is a filesystem issue that must be addressed by the administrator.

Rack Awareness
Rack Awareness deals with data locality. The main design goals of Hadoop MapReduce is to move
the computation to the data.
Three levels of data locality:
1. Data resides on the local machine (best).
2. Data resides on the same rack (better).
3. Data resides in a different rack (good).
When the YARN scheduler is assigning MapReduce containers to work as mappers, it will try to
place the container first on the local machine, then on the same rack, and finally on another rack.
In addition, the NameNode tries to place replicated data Blocks on multiple racks for improved fault
tolerance. In such a case, an entire rack failure will not cause data loss or stop HDFS from working.
Performance may be degraded.
HDFS can be made rack-aware by using a user-derived script that enables the master node to
map the network topology of the cluster.
NameNode High Availability
With early Hadoop installations, the NameNode was a single point of failure that could bring down
the entire Hadoop cluster. NameNode hardware often employed redundant power supplies and
storage to guard against such problems, but it was still susceptible to other failures. The solution was
to implement NameNode High Availability (HA) as a means to provide true failover service.

Figure 1.3: HDFS High Availability Design

Department of CSE Page 5

Module-1 BDA-15CS82

As shown in Figure 1.3, an HA Hadoop cluster has two (or more) separate NameNode machines.
Each machine is configured with exactly the same software. One of the NameNode machines is in
the Active state and the other is in the Standby state. Like a single NameNode cluster, the Active
NameNode is responsible for all client HDFS operations in the cluster. The Standby NameNode
maintains enough state to provide a fast failover.
To guarantee the file system state is preserved, both the Active and Standby NameNodes
receive block reports from the DataNodes. The Active node also sends all file system edits to a
quorum of Journal nodes. At least three physically separate JournalNode daemons are required,
because edit log modifications must be written to a majority of the JournalNodes. This design will
enable the system to tolerate the failure of a single JournalNode machine. The Standby node
continuously reads the edits from the JournalNodes to ensure its namespace is synchronized with that
of the Active node. In the event of an Active NameNode failure, the Standby node reads all
remaining edits from the JournalNodes before promoting itself to the Active state.
Apache ZooKeeper is used to monitor the NameNode health. Zookeeper is a highly available
service for maintaining small amounts of coordination data, notifying clients of changes in that data,
and monitoring clients for failures. HDFS failover relies on Zookeeper for failure detection and for
Standby to Active NameNode election.
HDFS NameNode Federation
The key benefits of NameNode Federation are as follows.
1. Namespace scalability: HDFS cluster storage scales horizontally without placing a burden on
the NameNode.

Figure 1.4: HDFS NameNode Federation example

Department of CSE Page 6

Module-1 BDA-15CS82

2. Better Performance: Addinf more NameNode to the cluster scales the file system read/write
operation throughout by sepserating the total namespace.
3. System isolation: Multiple NameNodes enable different categories of applications to be
distinguished and users can be isolated to different namespaces.
Figure 1.4 illustrates how HDFS NameNode Federation is accomplished. NameNode1 manages the
/research and /marketing namespaces and NameNode 2 manages the /data and /project namespaces.

HDFS Checkpoints and Backups

The NameNode stores the metadata of the HIDFS file system in a file called fsimage. File systems
modifications are written to an edits log file, and at startup the NameNode merges the edits into a
new fsimage. The Secondary-NameNode or CheckpointNode periodically fetches edits from the
NameNode merges them, and returns an updated fsimage to the NameNode.
An HDFS BackupNode is similar, but also maintains an up-to-date copy of the file system
namespace both in memory and on disk. Unlike a CheckpointNode, the BackupNode does not need
to download the fsimage and edits files from the active NameNode because it already has an up-to-
date namespace state in memory. A NameNode supports one Back-upNode at a time. No
CheckpointNodes may be registered if a Backup node is in use.

HDFS Snapshots
HDFS snapshots are similar to backups, but are created by administrators using the
hdfs dfs -snapshot command. HDFS snapshots are read-only point-in-time copies of the file system.
They offer the following features:
 Snapshots can be taken of a sub-tree of the file system or the entire file system.
 Snapshots can be used for data backup, protection against user errors, and disaster recoverv.
 Snapshot creation is instantaneous.
 Blocks on the DataNodes are not copied, because the snapshot files record the block list and
the file size. There is no data copying, although it appears to the user that there are duplicate
files.
 Snapshots do not adversely affect regular HDFS operations.

HDFS NFS Gateway

The HDFS NFS Gateway supports NFSv3 and enables HDFS to be mounted as part of the client's

Department of CSE Page 7

Module-1 BDA-15CS82

local file system. Users can browse the HDFS file system through their local file system that provide
an NFSv3 client compatible operating system. This feature offers users the following capabilities:
 Users can easily download/upload files from/to the HDFS file system to/from their local file
system.
 Users can stream data directly to HDFS through the mount point. Appending to a file is
supported, but random write capability is not supported.

HDFS User Commands

The following is a brief command reference that will facilitate navigation within HDFS. Brief HDFS

Command Reference
The preferred way to interact with HDFS in Hadoop version 2 is through the hdfs command. In
version 1 hadoop dfs command was used to manage file in HDFS.
Usage: hdfs [--config confdir] COMMAND
where COMMAND is one of:
dfs run a file system command on the file systems supported in Hadoop.
namenode –format format the DFS file system
secondarynamenode run the DFS secondary namenode
namenode run the DFS namenode
journalnode run the DFS journalnode
zkfc run the ZK Failover Controller daemon
datanode run a DFs datanode
dfsadmin run a DFS admin client
haadmin run a DFS HA admin client
fsck run a DFs file system checking utility
balancer run a cluster balancing utility
jmxget get JMX exported values from NameNode or DataNode.
Mover runs a utility to move block replicas across storage types
Oiv apply the offline fsimage viewer to an fsimage
oiv_legacy apply the offline fsimage viewer to an legacy fsimage
oev apply the offline edits viewer to an edits file
fetchdt fetch a delegation token from the NameNode
getconf get config values from configuration

Department of CSE Page 8

Module-1 BDA-15CS82

groups get the groups which users belong to

snapshotDiff diff two snapshots of a directory or diff the current directory contents with a
snapshot
lsSnapshottableDir list all snapshottable dirs owned by the current user Use -help to see options
portmap run a portmap service
nfs3 run an NFS version 3 gateway
cacheadmin configure the HDFS cache
crypto configure HDFS encryption zones
storagepolicies get all the existing block storage policies
version print the version
Most commands print help when invoked w/o parameters.

General HDFS Commands

The version of HDFS can be found from the version option. Examples in this section are run on the
HDFS version shown here:
$ hdfs version
Hadoop 2.6.0.2.2.4.2-2
Subversion [email protected]:hortonworks/hadoop.git-r
22a563ebe448969d07902aed869ac13c6522872
Compiled by jenkins on 2015-03-31T19:49Z
Compiled with protoc 2.5.0
From source with checksum b3481c2cdbe2d181f2621331926e267
This command was run using /usr/hdp/2.2.4.2-2/hadoop/hadoop-common-2.6.0.2.2.4.2-2.jar

HDFS provides a series of commands similar to those found in a standard POSIX file system.
A list of those commands can be obtained by issuing the following command. Several of these
commands will be highlighted here under the user account hdfs.
$ hdfs dfs

List Files in HDFS

To list the files in the root HDFS directory, enter the following command:
$hdfs dfs –ls /
To list the files in home directory, enter the following command:
$hdfs dfs –ls

Department of CSE Page 9

Module-1 BDA-15CS82

The same result can be obtained by issuing the following command:

$hdfs dfs –ls /user/hdfs

Make a Directory in HDFS

To make a directory in HDFS, use the following command, As with the -ls command when no path
is supplied,the user's home directory is used(e.g.,/users/hdfs).
$ hdfa dfs -mkdir stuff

Copy Files to HDFS

To copy a file from your current local directory into HDFS, use the following command. If a full
path is not supplied, your home directory is assumed. In this case, the file test is placed in the
directory stuff that was created previously.
$ hdfs dfs -put test stuff
The file transfer can be confirmed by using the -ls command:
$ hdfs dfs -ls stuff
Found 1 items
~rw-r--r-- 2 hdfs hdfs 12857 2015-05-29 13:12 stuff/test

Copy Files from HDFS

Files can be copied back to your local file system using the following command. In this case, the file
we copied into HDFS, test, will be copied back to the current local directory with the name test-local.
$ hdfs dfs -get stuff/test test-local

Copy Files within HDFS

The following command will copy a file in HDFS:
$ hdfs dfs -cp stuff/test test.hdfs

Delete a File within HDFS

The following command will delete the HDFS file test.hdfs that was created previously:
$ hdfs dfs -rm test.hdfs
Moved: 'hdfs://limulus:8020/user/hdfs/stuff/test' to trash at:
hdfs://limulus:8020/user/hdfs/.Trash/Current
Note that when the fs.trash.interval option is set to a non-zero value in core-site.xml, all deleted files
is moved to the user's Trash directory. This can be avoided by including the-skipTrash option.

Department of CSE Page 10

Module-1 BDA-15CS82

$ hdfs dfs -rm -skipTrash stuff/test

Deleted stuff/test

Delete a Directory in HDFS

The following command will delete the HDFS directory stuff and all its contents:
$hdfs dfs –rm –r –skiptrash stuff
Deleted stuff

Get an HDFS Status Report

$hdfs dfsadmin –report

Using HDFS in Programs

HDFS Java Application Example

When using Java, reading from and writing to Hadoop DFS is no different from the corresponding
operations with other file systems, The code is an example of reading, writing, and deleting files
from HDFS, as well as making directories
To be able to read from or write to HDFS, you need to create a Configuration object and pass
configuration parameters to it using Hadoop configuration files. The example assumes the Hadoop
configuration files are in /etc/hadoop/conf. If you do not assign the configuration objects to the local
Hadoop XML files,your HDFS operation will be performed on the local file system and not on the
HDFS.

Department of CSE Page 11

Module-1 BDA-15CS82

Department of CSE Page 12

Module-1 BDA-15CS82

Department of CSE Page 13

Module-1 BDA-15CS82

Department of CSE Page 14

Module-1 BDA-15CS82

The HadoopDFSFileReadWrite.java example can be compiled on Linux systems using the following
steps.
 First create directory to hold the classes:
$mkdir HDFSClient –classes
 Next compile the program using ‘hadoop classpath’ to ensure all the class path are available:
$java –cp ‘hadoop classpath’ –d HDFSClient-classes HDFSClient.java
 Finally, create a java archive file:
$jar –cvfe HDFSClient.jar org/myorg.HDFSClient –c HDFSClient-classes/
The program can be run to check for available options as follows:
$hadoop jar ./HDFSClient.jar
Usage: hdfsclient add/read/delete/mkdir [<local_path> <hdfs_path>]
A simple file copy from the local system to HDFS can be accomplished using the following
command:
$hadoop jar ./HDFSClient.jar add ./NOTES.tst /user/hdfs

HDFS C Application Example

HDFS can be used in C program by incorporating the Java Native Interface(JNI) based C application
programming interface(API) for Hadoop HDFS.

Department of CSE Page 15

Module-1 BDA-15CS82

The example can be build using the following steps:

The first step loads the Hadoop environment paths.
$ ./etc/hadoop/conf/hadoop-env.sh
The program is compiled using gcc and the following command line.
$ gcc hdfs-simple-test.c –I$HADOOP_LIB/include –I$JAVA_HOME/include –
L$HADOOP_LIB/lib –L$JAVA_HOME/jre/lib/amd64/server -1jvm –lhdfs –o hdfs-simple-test
The location of the run-time library path needs to be set with the following command:
$export LD_LIBRARY_PATH=
$LD_LIBRARY_PATH:$JAVA_HOME/jreamd64/server:$HADOOP_LIB/lib
The Hadoop class path needs to be set with the following command:
$export CLASSPATH=’hadoop classpath –glop’
The program can be run using the following.
$/hdfs-simple-test
The new file contents can be inspected using the hdfs dfs –cat command:
$hdfs dfs –cat /tmp/testfile.txt
Hello, World!

Department of CSE Page 16

Module-1 BDA-15CS82

Chapter 2: Running Example Programs and Benchmarks

Running MapReduce Examples

To run an existing Map Reduce Example, it should be located first

For Hadoop version 2.6.0, the examples files are located in the following directory

/opt/hadoop-2.6.0/share/hadoop/mapreduce/

In other versions

/usr/lib/hadoop-mapreduce/

or other location

To find the exact location of the example jar file

$ find / -name “hadoop-mapreduce-examples*.jar”– print

Listing Available Examples

$ yarn jar $ HADOOP EXAMPLES/hadoop-mapreduce- examples.jar

The following is a list of the included jobs in the examples JAR file.
 aggregatewordcount: An Aggregate-based map/reduce program that counts
the words in the input files.
 aggregatewordhist: An Aggregate-based map/reduceprogram that computes the histogram
of the words in the input files.
 bbp: A map/reduce program that usesBailey-Borwein-Plouffe to compute the exact digits of
pi.
 dbcount: An example job that counts the pageviewcounts from a database.
 distbbp: A map/reduce program that uses a BBP-typeformula to compute the exact bits of
pi.
 grep: A map/reduce program that counts the matches toa regex in the input.
 join: A job that effects a join over sorted, equally partitioned data sets.
 multifilewc: A job that counts words from several files.
 pentomino: Amap/reduce tile laying program to find solutions to pentomino

Department of CSE Page 17

Module-1 BDA-15CS82

problems.
 pi: A map/reduce program that estimates pi using a quasi-Monte Carlo method.
 randomtextwriter: A map/reduce program that writes 10 GB ofrandom textual data per
node.
 randomwriter: Amap/reduce program that writes 10 GB of random data per node.
 secondarysort: An example defining a secondary sort to thereduce.
 sort: A map/reduce program that sorts the data written by therandom writer.
 sudoku: A Sudokusolver.
 teragen: Generate data for the terasort.
 terasort: Run the terasort.
 teravalidate: Check the results of the terasort.
 wordcount: A map/reduce program that counts the wordsin the input files.
 wordmean: A map/reduce program that counts theaverage length of the words in the
input files.
 wordmedian: A map/reduce program that counts themedian length of the words in the input
files.
 wordstandarddeviation: A map/reduce program that counts the standard deviation of
the length of the words in the input files.
Run Sample MapReduce Examples
To test your installation, run the sample “pi” program that calculates the value of pi using a quasi-
Monte Carlo method and MapReduce. Change to user hdfs and run the following:

# su - hdfs

$ cd /opt/yarn/hadoop-2.2.0/bin $ export
YARN_EXAMPLES=/opt/yarn/hadoop-
2.2.0/share/hadoop/mapreduce

$ ./yarn jar

$YARN_EXAMPLES/hadoop-mapreduce-examples- 2.2.0.jar pi 16 1000000

 If the program worked correctly, the following should be displayed at the end of the
program output stream:

 Estimated value of Pi is 3.14250000000000000000

Department of CSE Page 18

Module-1 BDA-15CS82

 The output will contain

Department of CSE Page 19

Module-1 BDA-15CS82

Using the web GUI to monitor Examples

• Now, we will look at the YARN web GUI to monitor the examples.
• You can monitor the application submission ID, the user who submitted the
application, the name of the application, the queue in which the application is
submitted, the start time and finish time in the case of finished applications, and the
final status of the application, using the ResourceManager UI.
• The ResourceManager web UI differs from the UI of the Hadoop 1.x versions.
• The following screenshot shows the information we could get from the YARN
web UI (http://localhost:8088).

Application Status Window

Hadoop RUNNING Applications web GUI for the pi example

Department of CSE, HKBKCE Page 20

Module-1 BDA-15CS82

Hadoop YARN application status for the pi example

Hadoop YARNMapReduce job progress

Department of CSE, HKBKCE Page 21

Module-1 BDA-15CS82

Tracking URL leads to overall progress of mappers and reducers

Clicking on job….

Hadoop YARN ResourceManager nodes status window

Hadoop YARN ApplicationManager for Mapreduce application

Department of CSE, HKBKCE Page 22

Module-1 BDA-15CS82

Hadoop YARN NodeManager for n0 job summary

Hadoop YARN completed MapReduce job summary

Department of CSE, HKBKCE Page 23

Module-1 BDA-15CS82

Hadoop YARN application summary page

Running Basic Hadoop Benchmarks

 Many Hadoop benchmarks can provide insight into cluster performance.

 The best benchmarks are always those that reflect real application
performance.

 The two benchmarks discussed in this

section, terasort and TestDFSIO, provide a good sense of how well your Hadoop
installation is operating and can be compared with public data published for other
Hadoop systems.
Running the Terasort Test

 The terasort benchmark sorts a specified amount of randomly generated data. This
benchmark provides combined testing of the HDFS and MapReduce layers of a
Hadoop cluster.

 A full terasortbenchmark run consists of the followingthree steps:

o Generating the input data via teragen program.

o Running the actual terasort benchmark on the input data.

Department of CSE, HKBKCE Page 24

Module-1 BDA-15CS82

o Validating the sorted output data via the teravalidate program.

o Run teragen to generate rows of random data to sort.

$ yarn jar
$HADOOP_EXAMPLES/hadoop-mapreduce-examples.jar teragen
500000000 /user/hdfs/TeraGen-50GB

1. Run terasort to sort the database.

$ yarn jar
$HADOOP_EXAMPLES/hadoop-mapreduce-examples.jar terasort /user/hdfs/TeraGen-
50GB /user/hdfs/TeraSort-50GB

2. Run teravalidate to validate the sort.

$ yarn jar $HADOOP_EXAMPLES/hadoop-mapreduce-examples.jar teravalidate
/user/hdfs/TeraSort-50GB
/user/hdfs/TeraValid-50GB
For best performance, the actual terasort benchmark should be run with a replication
factor of 1. In addition, the default number of terasort reducer tasks is set to 1. Increasing
the number of reducers often helps with benchmark performance
• For example, the following command will instruct terasort to use four reducer tasks:
$ yarn jar $HADOOP_EXAMPLES/hadoop-mapreduce- examples.jar terasort -
Dmapred.reduce.tasks=4
/user/hdfs/TeraGen-50GB /user/hdfs/TeraSort-50GB

• The following command will perform the cleanup for the previous
example:
• $ hdfs dfs -rm -r -skipTrash Tera*
Running the TestDFSIO Benchmark

• Hadoop also includes an HDFS benchmark application called TestDFSIO.

The TestDFSIO benchmark is a read and write test for HDFS.

• the TestDFSIO benchmark is part of

the hadoop-mapreduce-client-jobclient.jar

Department of CSE, HKBKCE Page 25

Module-1 BDA-15CS82

• The steps to run TestDFSIO are as follows:

1. Run TestDFSIO in write mode and create data.

• $ yarn jar $HADOOP_EXAMPLES/hadoop-mapreduce-client-
jobclient-tests.jar TestDFSIO -write -nrFiles 16 -fileSize 1000

Example results are as follows (date and time prefix removed).

• fs.TestDFSIO: ----- TestDFSIO ----- : write fs.TestDFSIO: Date &

time: Thu May 14 10:39:33 EDT 2015

• fs.TestDFSIO: Number of files: 16

• fs.TestDFSIO: Total MBytes processed: 16000.0
• fs.TestDFSIO: Throughput mb/sec: 14.890106361891005
fs.TestDFSIO: Average IO rate mb/sec: 15.690713882446289
fs.TestDFSIO: IO rate std deviation: 4.0227035201665595 fs.TestDFSIO:
Test exec time sec: 105.631
2. Run TestDFSIO in read mode.
• $ yarn jar $HADOOP_EXAMPLES/hadoop-mapreduce- client-jobclient-tests.jar
TestDFSIO -read -nrFiles 16 -fileSize 1000

Example results are as follows (date and time prefix removed). The large standard
deviation is due to the placement of tasksin the cluster on a small four-node cluster.
• fs.TestDFSIO: ----- TestDFSIO ------- read
• fs.TestDFSIO: Date & time: Thu May 14 10:44:09 EDT 2015
fs.TestDFSIO: Number of files: 16
• fs.TestDFSIO: Total MBytes processed: 16000.0 fs.TestDFSIO: Throughput
mb/sec: 32.38643494172466 fs.TestDFSIO: Average IO rate mb/sec:
58.72880554199219 fs.TestDFSIO: IO rate std deviation: 64.60017624360337
fs.TestDFSIO: Test exec time sec: 62.798
3. Clean up the TestDFSIO data.

• $ yarn jar $HADOOP_EXAMPLES/hadoop-mapreduce- client-jobclient-

tests.jar TestDFSIO -clean

Department of CSE, HKBKCE Page 26

Module-1 BDA-15CS82

Managing Hadoop MapReduce Jobs

• Hadoop MapReduce jobs can be managed using the mapred job command.
• The most important options for this command in terms of the examples and
benchmarks are -list, -kill, and -status.
• In particular, if you need to kill one of the examples or benchmarks, you can use the
mapred job -listcommand to find the job-id and then use mapred job -kill <job-id> to
kill the job across the cluster.
• MapReduce jobs can also be controlled at the application level with the yarn
applicationcommand
• The possible options for mapred job are as follows:
$ mapred job Usage: CLI <command> <args>
[-submit <job-file>]
[-status <job-id>]
[-counter <job-id> <group-name> <counter-name>]
[-kill <job-id>]
[-set-priority <job-id> <priority>].
Valid values for priorities are: VERY_HIGH HIGH NORMAL LOW
VERY_LOW
[-events <job-id> <from-event-#> <#-of-events>]
[-history <jobHistoryFile>]
[-list [all]]
[-list-active-trackers]
[-list-blacklisted-trackers]
[-list-attempt-ids <job-id> <task-type> <task-state>].
Valid values for <task-type> are REDUCE MAP. Valid values for <task- state>
are running, completed
-conf <configuration file> specify an application configuration file

-D <property=value> use value for given property

-fs <local|namenode:port> specify a namenode

-jt <local|resourcemanager:port> specify a ResourceManager

Department of CSE, HKBKCE Page 27

Module-1 BDA-15CS82

-files <comma separated list of files> specify comma separated files to be copied to the
map reduce cluster

-libjars <comma separated list of jars> specify comma separated jar files to include in the
classpath.

-archives <comma separated list of archives> specify comma separated archives to be

unarchived on the compute machines.

The general command line syntax is bin/hadoop command [genericOptions]

[commandOptions]

Chapter 3: Hadoop Map Reduce Framework

The MapReduce Model

The MapReduce computation model provides a very powerful tool for many applications and
is more common than most users realize.
There are two stages:
 A mapping stage
 A reducing stage
In mapping stage, a mapping procedure is applied to input data. The map is usually some
kind of filter or sorting process.
Example: assume you need to count how many times the name “kutuzou” appears in the
novel War and Peace. One solution is to gather 20 friends and give them each a section of the
book to search. This step is the map stage.
The reduce phase happens when everyone is done counting and you sum the total as your
friends tell you their counts.
Now consider how this same process could be accomplished using simple *nix command-line
tools. The following grep command applies a specific map to a text file.
$ grep" Kutuzov" war-and-peace,txt
This command searches for the word Kutuzov in a text file called war-and-peace.txt.
Each match is reported as a single line of text that contains the search term. The actual text
file is a 3.2MB text dump of the novel War and Peace. The search term, Kutuzov, is a
character in the book.

Department of CSE, HKBKCE Page 28

Module-1 BDA-15CS82

If we ignore the grep count (-c) option for the moment, we can reduce the number of
instances to a single number (257) by sending (piping) the results of grep into wc-1
$ grep" Kutuzov "war-and-peace.txt|wc-1
257
Though not strictly a MapReduce process, this idea is quite similar to and much faster
than the manual process of counting the instances of Kutuzov in the printed book. The
analogy can be taken a bit further by using the two simple (and naive) shell scripts shown
below. We can perform the same operation and tokenize both the Kutuzov and Petersburg
strings in the text:
$ cat war-and-peace.txt|./mapper.sh |./reducer.sh
Kutuzov, 315
Petersburg, 128
Notice that more instances of Kutuzov have been found. The mapper inputs a text file
and then outputs data in a (key,value) pair (token-name,count) format. Strictly speaking, the
input to the script is the file and the keys are Kutuzov and Petersburg. The reducer script
takes these key-value pairs and combines the similar tokens and counts the total number of
instances. The result is a new key-value pair (token-name, sum).

Simple Mapper Script

#!/bin/bash
while read line ; do
for token in $line; do
if ["$token"="Kutuzov"];then
echo "Kutuzov,1"
elif ["$token" ="Petersburg" ] ; then
echo "Petersburg,1"
fi
done
done
Simple Reducer Script
#!/bin/bash
kcount=0
pcount=0
while read line ; do

Department of CSE, HKBKCE Page 29

Module-1 BDA-15CS82

if ["$line”= "Kutuzov,1" ] ; then

let kcount=kcount+1
elif ["$line" = "Petersburg,1" ] ; then
let pcount=pcount+1
fi
done
echo "Kutuzov,$kcount”
echo "Petersburg,$pcount"

The MapReduce process can be described as follows.

The mapper and reducer functions are both defined with respect to data structured in
(key,value)pairs. The mapper takes one pair of data with a type in one data domain, and
returns a list of pairs in a different domain:
Map (key1,valuel) list(key2,value2)
The reducer function is then applied to each key-value pair, which in turn produces a
collection of values in the same domain:
Reduce(key2,list(value2))list(value3)
Each reducer call typically produces either one value (value3)or an empty response.
Thus, the MapReduce framework transforms a list of(key, value)pairs into a list of values.
The MapReduce model is inspired by the map and reduces functions commonly used
in many functional programming languages. The functional nature of Map- Reduce has some
important properties:
 Data flow is in one direction (map to reduce).It is possible to use the output of a
reduce step as the input to another MapReduce process.
 As with functional programming, the input data are not changed. By applying the
mapping and reduction functions to the input data new data are produced.
 Because there is no dependency on how the mapping and reducing functions are
applied to the data,the m apper and reducer data flow can be implemented in any
number of ways to provide better performance.
Distributed (parallel) implementations of MapReduce enable large amounts of data to be
analyzed quickly. In general, the mapper process is fully scalable and can be applied to any
subset of the input data. Results from multiple parallel mapping functions are then combined
in the reducer phase.

Department of CSE, HKBKCE Page 30

Module-1 BDA-15CS82

Apache Hadoop MapReduce will try to move the mapping tasks to the server that
contains the data slice. Results from each data slice are then combined in the reducer step.
HDFS is not required for Hadoop MapReduce,however.A sufficiently fast parallel file
system can be used in its place. In these designs, each server in the cluster has access to a
high-performance parallel file system that can rapidly provide any data slice. These designs
are typically more expensive than the commodity servers used for many Hadoop clusters.

MapReduce Parallel Data Flow

From a programmer's perspective, the MapReduce algorithm is fairly simple. The

programmer must provide a mapping function and a reducing function. The basic steps are as
follows:
1. Input Splits: HDFS distributes and replicates data over multiple servers. The default data
chunk or block size is 64MB. Thus, a 500MB file would be broken into 8 blocks and
written to different machines in the cluster. The data are also replicated on multiple
machines. These data slices are physical boundaries determined by HDFS and have
nothing to do with the data in the file. Also, while not considered part of the Map-Reduce
process, the time required to load and distribute data throughout HDFS servers can be
considered part of the total processing time. The input splits used by MapReduce are
logical boundaries based on the input data. For example, the split size can be based on the
number of records in a file or an actual size in bytes. Splits are almost always smaller than
the HDFS block size. The number of splits corresponds to the number of mapping
processes used in the map stage.
2. Map Step: The mapping process is where the parallel nature of Hadoop comes into play.
For large amounts of data, many mappers can be operating at the same time. The user
provides the specific mapping process. MapReduce will try to execute the mapper on the
machines where the block resides. Because the file is replicated in HDFS, the least busy
node with the data will be chosen. If all nodes holding the data are too busy, MapReduce
will try to pick a node that is closest to the node that hosts the data block (a characteristic
called rack-awareness). The last choice is any node in the cluster that has access to
HDFS.

Department of CSE, HKBKCE Page 31

Module-1 BDA-15CS82

3. Combiner Step: It is possible to provide an optimization or pre-reduction as part of the

map stage where key-value pairs are combined prior to the next stage. The combiner
stage is optional.
4. Shuffle Step: Before the parallel reduction stage can complete, all similar keys must be
combined and counted by the same reducer process. Therefore, results of the map stage
must be collected by key-value pairs and shuffled to the same reducer process. If only a
single reducer process is used, the shuffle stage is not needed.
5. Reduce Step: The final step is the actual reduction. In this stage, the data reduction is
performed as per the programmer's design. The reduce step is also optional. The results
are written to HDFS. Each reducer will write an output file.
For example, a MapReduce job running four reducers will create files called part-0000,
part-0001, part-0002, and part-0003.
Figure 1.5 is an example of a simple Hadoop MapReduce data flow for a word count
program. The map process counts the words in the split, and the reduce process calculates
the total for each word. stages are up to the programmer. The MapReduce data flow
shown in Figure 1.5 is the same regardless of the specific map and reduces tasks.

Figure 1.5: Apache Hadoop parallel MapReduce data flow

The input to the MapReduce application is the following file in HDFS with three lines of text.
The goal is to count the number of times each word is used.
See spot run
Run spot run
See the cat

Department of CSE, HKBKCE Page 32

Module-1 BDA-15CS82

The first thing MapReduce will do is create the data splits. For simplicity, each
line will be one split. Since each split will require a map task, there are three mapper
processes that count the number of words in the split. On a cluster, the results of each
map task are written to local disk and not to HDFS. Next, similar keys need to be
collected and sent to a reducer process. The shuffle step requires data movement and
can be expensive in terms of processing time. Depending on the nature of the application, the
amount of data that must be shuffled throughout the cluster can vary from small to large.
Once the data have been collected and sorted by key, the reduction step can begin. It
is not necessary and not normally recommended to have a reducer for each key-value pair as
shown inFigure1.5. The number of reducers is a tunable option for many applications. The
final step is to write the output to HDFS. A combiner step enables some pre-reduction of the
map output data. For instance, in the previous example, one map produced the following
counts:
(run,1)
(spot,1)
(run,1)
As shown in Figure 1.6, the count for run can be combined into run, & before the
shuffle. This optimization can help minimize the amount of data transfer needed for
the shuff phase.

Figure 1.6: Adding a combiner process to the map step in MapReduce

The Hadoop YARN resource manager and the MapReduce framework determine the actual
placement of mappers and reducers. The MapReduce framework will try to place the map
task as close to the data as possible. It will request the placement from the YARN scheduler
but may not get the best placement due to the load on the cluster. In general, nodes can run

Department of CSE, HKBKCE Page 33

Module-1 BDA-15CS82

both mapper and reducer tasks. Indeed, the dynamic nature of YARN enables the work
containers used by completed map tasks to be returned to the pool of available resources.
Figure 1.7 shows a simple three-node MapReduce process. Once the mapping is
complete, the same nodes begin the reduce process. The shuffle stage makes sure the
necessary data are sent to each mapper. Also note that there is no requirement that all the
mappers complete at the same time or that the mapper on a specific node be complete before
a reducer is started. Reducers can be set to start shuffling based on a threshold of percentage
of mappers that have finished.
Finally, although the examples are simple in nature, the parallel MapReduce
algorithm can be scaled up to extremely large data sizes.

Fault Tolerance and Speculative Execution

One of the most interesting aspects of parallel MapReduce operation is the strict control of
data flow throughout the execution of the program. For example, mapper processes do not
exchange data with other mapper processes, and data can only go from mappers to reducers-
not the other direction. The confined data flow enables MapReduce to operate in a fault-
tolerant fashion.
The design of MapReduce makes it possible to easily recover from the failure of one
or many map processes. For example, should a server fail, the map tasks that were running on
that machine could easily be restarted on another working server because there is no
dependence on any other map task. In functional language terms, the map tasks "do not share
state" with other mappers. Of course, the application will run more slowly because work
needs to be redone, but it will complete.

Speculative Execution
One of the challenges with many large clusters is the inability to predict or manage
unexpected system bottlenecks or failures. In theory, it is possible to control and monitor
resources so that network traffic and processor load can be evenly balanced; in practice,
however, this problem represents a difficult challenge for large systems. Thus, it is possible
that a congested network, slow disk controller, failing disk, high processor load, or some
other similar problem might lead to slow performance without anyone noticing.
When one part of a MapReduce process runs slowly, it ultimately slows down
everything else because the application cannot complete until all processes are finished. The

Department of CSE, HKBKCE Page 34

Module-1 BDA-15CS82

nature of the parallel MapReduce model provides an interesting solution to this problem. It is
possible to start a copy of a running map process without disturbing any other running
mapper processes. For example, suppose that as most of the map tasks are coming to a close,
the ApplicationMaster notices that some are still running and schedules redundant copies of
the remaining jobs on less busy or free servers. Should the secondary processes finish first,
the other first processes are then terminated (or vice versa).This process is known as
speculative execution.

Hadoop MapReduce Hardware

The capability of Hadoop MapReduce and HDFS to tolerate server-or even whole rack-
failures can influence hardware designs. The use of commodity servers for Hadoop clusters
has made low-cost, high-availability implementation of Hadoop possible for many data
centers. The Apache Hadoop philosophy seems to assume servers will always fail and takes
steps to keep failure from stopping application progress on a cluster.
The use of server nodes for both storage (HDFS) and processing (mappers, reducers)
is somewhat different from the traditional separation of these two tasks in the data center. It is
possible to build Hadoop systems and separate the roles (discrete age and processing nodes).
However, a majority of Hadoop systems use the general approach where servers enact both
roles. Another interesting feature of dynamic MapReduce execution is the capability to
tolerate dissimilar servers. That is, old and new hardware can be used together. Of course,
large disparities in performance will the faster systems, but the dynamic nature of
MapReduce execution will still work effectively on such systems.

Chapter 4: MapReduce Programming

In this chapter we will discuss about

 The classic Java WordCount program for Hadoop
 A python WordCount Application using the Hadoop streaming interface is introduced
 The Hadoop pipes interface is used to run a C++ version of wordcount
 An example of MapReduce chaining using Hadoop Grep example
 Strategies for MapReduce debugging are presented

Compiling and Running the Hadoop WordCount Example

Department of CSE, HKBKCE Page 35

Module-1 BDA-15CS82

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
Importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>
{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context ) throws IOException,
InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws

IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values)
{ sum += val.get(); }
result.set(sum);

Department of CSE, HKBKCE Page 36

Module-1 BDA-15CS82

context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new
Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
The framework views the input to the jobs as a set of keyvalue pairs
The MapReduce job proceeds as follows
(input) <k1,v1>--map--<k2,v2>--combine-<k2,v2>--reduce -<k3,k4> (output)
Two inputs
Hello World Bye World
Hello Hadoop Goodbye Hadoop
The WordCount application is quite straight-forward.
public void map(Object key, Text value, Context context) throws IOException,
InterruptedException {
StringTokenizer itr = new tringTokenizer(value.toString());
while (itr.hasMoreTokens())
{ word.set(itr.nextToken());
context.write(word, one);
}}
key-value pair of < <word>, 1>.
For the given sample input the first map emits:
For the given sample input the first map emits:
< Hello, 1>

Department of CSE, HKBKCE Page 37

Module-1 BDA-15CS82

< World, 1>

< Bye, 1>
< World, 1>
The second map emits:
< Hello, 1>
< Hadoop, 1>
< Goodbye, 1>
< Hadoop, 1>
• WordCount also specifies a combiner.
• job.setCombinerClass(IntSumReducer.class);
The output of the first map:
< Bye, 1>
< Hello, 1>
< World, 2>
The output of the second map:
< Goodbye, 1>
< Hadoop, 2>
< Hello, 1>
public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws
IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values)
{ sum += val.get(); }
result.set(sum);
context.write(key, result);
}
The Reducer implementation, via the reduce method just sums up the values, which are the
occurrence counts for each key
the output of the job is:
< Bye, 1>
< Goodbye, 1>
< Hadoop, 2>
< Hello, 2>
< World, 2>

Department of CSE, HKBKCE Page 38

Module-1 BDA-15CS82

To compile and run the program from the command line, perform the following steps
1. Make a local wordcount-classes directory
$ mkdir wordcount-classes
2. Compile the wordcount.java program using the hadoop classpath
$ java –cp ‘hadoop classpath’ d wordcount-classes wordcount.java
3.The jar file is created
$ jar –cvf wordcount.jar –c wordcount-classes/
4. To run the program,create an input directory in HDFS and place a text file in the new
directory
$hdfs dfs –mkdir war-and-peace-input
$ hdfs dfs –put war-and-peace.txt war-and-peace-input
5.Run the wordcount application
$ hadoop jar wordcount.jar wordcount war-and-peace-input war- and –peace output

Using the Streaming Interface

The Apache Hadoop steaming interface enables program to use the MapReduce engine.
The streams interface will work with any program that can read and write to stdin and stdout.
When working in Hadoop streaming mode the mapper and reducer are created by user.
This approach does have the advantages that the mapper and the reducer can be easily tested
from the command line.
In this example a Python mapper and reducer will be used.
Python mapper script(mapper.py)
#!/usr/bin/env python
import sys
# input comes from STDIN (standard input)
for line in sys.stdin:
# remove leading and trailing whitespace
line = line.strip()
# split the line into words
words = line.split()
# increase counters
for word in words:
# write the results to STDOUT (standard output);

Department of CSE, HKBKCE Page 39

Module-1 BDA-15CS82

# what we output here will be the input for the

# Reduce step, i.e. the input for reducer.py
# # tab-delimited; the trivial word count is 1
print '%s\t%s' % (word, 1)

Python Reducer Script(reducer.py)

#!/usr/bin/env python
from operator import itemgetter
import sys
current_word = None
current_count = 0
word = None
# input comes from STDIN
for line in sys.stdin:
# remove leading and trailing whitespace
line = line.strip()
# parse the input we got from mapper.py
word, count = line.split('\t', 1)
# convert count (currently a string) to int
try:
count = int(count)
except ValueError:
# count was not a number, so silently
# ignore/discard this line
continue
# this IF-switch only works because Hadoop sorts map output
# by key (here: word) before it is passed to the reducer
if current_word == word:
current_count += count
else:
if current_word:
# write result to STDOUT
print '%s\t%s' % (current_word, current_count)
current_count = count

Department of CSE, HKBKCE Page 40

Module-1 BDA-15CS82

current_word = word
# do not forget to output the last word if needed!
if current_word == word:
print '%s\t%s' % (current_word, current_count)

The operations of the mapper.py script can be observed by running the command as shown in
the following:
$ echo "foo foo quux labs foo bar quux" | mapper.py
foo 1
foo 1
quux 1
labs 1
foo 1 bar 1
quux 1

Piping the results of the map into the sort command can create a simulated shuffle phase:
echo "foo foo quux labs foo bar quux“ |mapper.py | sort -k1,1
Bar 1
foo 1
foo 1
Foo 1
labs 1
quux 1
quux 1

$ echo "foo foo quux labs foo bar quux“ |mapper.py | sort –k1,1|./reducer.py
Bar 1
Foo 1
Labs 1
quux 1
• The output directory is removed from any previous test runs:
$hdfs dfs –rm –r –skiptrash war-and-peace-output
• The following command line will use the mapper.py and reducer.py to do a word count
on the input file.

Department of CSE, HKBKCE Page 41

Module-1 BDA-15CS82

$hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar

-file ./ mapper.py \
-mapper ./ mapper.py \
-file ./reducer.py -reducer ./reducer.py \
-input war-and-peace-input/ war-and-peace,txt \
-output war-and-peace-output

Using the Pipes Interface

• Pipes are a library that allows C++ source code to be used for mapper and reducer code.
• The pipes framework on each machine assigned to job will start an instance of C++
program. So the executable must be placed in HDFS prior to use.
#include "hadoop/Pipes.hh"
#include "hadoop/TemplateFactory.hh"
#include "hadoop/StringUtils.hh“
class WordCountMap: public HadoopPipes::Mapper {
public:
WordCountMap(HadoopPipes::TaskContext& context){}
void map(HadoopPipes::MapContext& context) {
std::vector<std::string> words =
HadoopUtils::splitString(context.getInputValue(), " ");
for(unsigned int i=0; i < words.size(); ++i) {
context.emit(words[i], "1");
}
}
};
class WordCountReduce: public HadoopPipes::Reducer {
public:
WordCountReduce(HadoopPipes::TaskContext& context){}
void reduce(HadoopPipes::ReduceContext& context) {
int sum = 0;
while (context.nextValue()) {
sum += HadoopUtils::toInt(context.getInputValue());

Department of CSE, HKBKCE Page 42

Module-1 BDA-15CS82

}
context.emit(context.getInputKey(), HadoopUtils::toString(sum));
}
};
int main(int argc, char *argv[]) {
return HadoopPipes::runTask(HadoopPipes::TemplateFactory<WordCountMap,
WordCountReduce>());
}
• The program can be compiled with the following line
$ g++ wordcount.cpp –o wordcount –L$HADOOP_HOME/lib/native/
Create the war-and-peace-input directory and move the file into HDFS:
$ hdfs dfs –mkdir war-and-peace-input
$ hdfs dfs –put war-and-peace.txt war-and-peace-input
The executable must be placed into HDFS so YARN can find the program.
$ hdfs dfs –put wordcount bin
$ hdfs dfs –rm –r –skiptrash war-and-peace-output
To run the program, enter the following line
$ mapred pipes \
-D hadoop.pipes.java.recordreader=true \
-D hadoop.pipes.java.recordwriter=true \
Input war-and-peace-input \
Output war-and-peace-output \
-program bin/wordcount

Compiling and Running the Hadoop Grep Chaining Example

The Hadoop Grep.java example extracts matching strings from text files and counts how
many time they occured. The command works different than the Unix grep call: it doesn't
display the complete matching line, but only the matching string, so in order to display lines
matching "foo", use .*foo.* as a regular expression.
The program runs two map/reduce jobs in sequence.
 The first job counts how many times a matching string occured and
 The second job sorts matching strings by their frequency and stores the output in a
single output file.

Department of CSE, HKBKCE Page 43

Module-1 BDA-15CS82

Department of CSE, HKBKCE Page 44

Module-1 BDA-15CS82

 Each mapper of the first job takes a line as input and matches the user-provided regular
expression against the line. It extracts all matching strings and emits (matching string, 1)
pairs.

Department of CSE, HKBKCE Page 45

Module-1 BDA-15CS82

 The second job takes the output of the first job as input. The mapper is an inverse map, while
the reducer is an indentity reducer. The number of reducers is one, so the output is stored in
one file, and it is sorted by the count in a descending order. The output file is text, each line of
which contains count and a matching string. 
The following describes how to compile and run the Grep.java example. The steps are:
1. Create a directory for the application classes as follows:
$mkdir Grep_classes
2. Compile the WordCount.java program using following line
$ java –cp ‘hadoop classpath’ –d Grep_classes Grep.java
3. Create a java archive using the following command:
$ jar –cvf Grep.jar –C Grep_classes/
If needed, create the war-and-peace-input directory and move the file into HDFS:
$ hdfs dfs –mkdir war-and-peace-input
$ hdfs dfs –put war-and-peace.txt war-and-peace-input
Make sure the output directory has been removed by issuing the following command:
$ hdfs dfs –rm –r –skiptrash war-and-peace-output

Debugging MapReduce
The applications on a simpler system with smaller data sets.
 The errors on these systems are much easier to locate and track.
 If applications can run successfully on a single system with a subset of real data, then
running in parallel should be a simple task because the MapReduce algorithm is
transparently scalable.
Listing, Killing, and Job Status
The most import options are –list, -kill, and –status. The yarn application command can be
used to control all applications running on the cluster.
Hadoop Log Management
The MapReduce logs provide a comprehensive listing of both mappers and reducers. The
actual log output consists of three files- stdout, stderr and syslog.
There are two modes for log storage.
 The first method is to use log aggregation. In this mode, logs are aggregated in HDFS
and can be displayed in the YARN ResourceManager user interface or examined with
the yarn logs command.

Department of CSE, HKBKCE Page 46

Module-1 BDA-15CS82

 If log aggregation is not enabled, the logs will be placed locally on the cluster nodes
on which the mapper or reducer ran. The location of the unaggregated local logs is
given by the yarn.nodemanager.log-dirs property in the yarn-site.xml file. Without log
aggregation, the cluster nodes used by the job must be noted and then the log files
must be obtained directly from the nodes. Log aggregatio is highly recommended.

Enabling YARN Log Aggregtion

If Apache Hadoop was installed from the official Apache Hadoop sources, the following
settings will ensure log aggregation is turned on for the system. For example, it you are using
Apache Ambari, check under the YARN Service Configs tab for yarn.log.aggregation-enable.
The default setting is enabled.
To enable log aggregation, follow these steps:
1. Create the following directory in HDFS:
$ hdfs dfs –mkdir –p /yarn/logs
$ hdfs dfs –chown –R yarn:hadoop /yarn/logs
$ hdfs dfs -chown –R g+rw /yarn/logs
2. Add the following properties in the yarn-site.xml and restart all YARN services on
all nodes.
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/yarn/logs</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

Command-Line Log Viewing

The yarn logs command enables the logs to be easily viewd together without having
to hunt for individual log files on the cluster nodes. The options to yarn logs are as
follows:
$yarn logs
Retrieve logs for completed YARN applications

Department of CSE, HKBKCE Page 47

Module-1 BDA-15CS82

For example, after running the pi example program, the logs can be examined as
follows:
$ hadoop jar %HADOOP_EXAMPLES/hadoop-mapreduce-examples.jar pi 16
100000

The applicationId will start with application_ and appear under the Application-id
column.
$yarn application –list –appStates FINISHED

Next, run the following command to produce a dump of all logs for that application.
$yarn logs –application application_1432667013445_0001 >AppOut

Finally, the results for a single container can be found by entering this line:
$yarn logs –application application_1432667013445_0001 –containerId
container_1432667013445_0001_01_000023 –nodeAddress n1:45454|more

Department of CSE, HKBKCE Page 48