Acropolis Institute of Technology &

Research, Indore
www.acropolis.in
Data Analytics
By: Mr. Ronak Jain
Table of Contents
UNIT-III:
PROCESSING BIG DATA: Integrating disparate data stores, Mapping data to the programming framework, Connecting
and extracting data from storage, Transforming data for processing, subdividing data in preparation for Hadoop Map
Reduce.

December 13, 2023 3

Hadoop Distributed File
System (HDFS)
Google search engines

1998

2013
Hadoop’s Developers

2005: Doug Cutting and Michael J. Cafarella developed

Hadoop to support distribution for the Nutch search
engine project.
Doug Cutting
The project was funded by Yahoo.

2006: Yahoo gave the project to Apache

Software Foundation.
Google Origins

2003

2004

2006
Some Hadoop Milestones
• 2008 - Hadoop Wins Terabyte Sort Benchmark (sorted 1 terabyte
of data in 209 seconds, compared to previous record of 297 seconds)

• 2009 - Avro and Chukwa became new members of Hadoop

Framework family

• 2010 - Hadoop's Hbase, Hive and Pig subprojects completed, adding

more computational power to Hadoop framework

• 2011 - ZooKeeper Completed

• 2013 - Hadoop 1.1.2 and Hadoop 2.0.3 alpha.

- Ambari, Cassandra, Mahout have been added
Hadoop Framework Tools
 Hadoop’s Architecture

• Distributed, with some centralization

• Main nodes of cluster are where most of the computational

power and storage of the system lies

• Main nodes run TaskTracker to accept and reply to MapReduce

tasks, and also DataNode to store needed blocks closely as
possible

• Central control node runs NameNode to keep track of HDFS

directories & files, and JobTracker to dispatch compute tasks to
TaskTracker

• Written in Java, also supports Python and Ruby

Hadoop’s Architecture
Hadoop’s Architecture
• Hadoop Distributed Filesystem

• Tailored to needs of MapReduce

• Targeted towards many reads of filestreams

• Writes are more costly

• High degree of data replication (3x by default)

• No need for RAID on normal nodes

• Large blocksize (64MB)

• Location awareness of DataNodes in network

Hadoop’s Architecture
NameNode:

• Stores metadata for the files, like the directory structure of a

typical FS.

• The server holding the NameNode instance is quite crucial,

as there is only one.

• Transaction log for file deletes/adds, etc. Does not use

transactions for whole blocks or file-streams, only metadata.

• Handles creation of more replica blocks when necessary after

a DataNode failure
Hadoop’s Architecture
DataNode:

• Stores the actual data in HDFS

• Can run on any underlying filesystem (ext3/4, NTFS, etc)

• Notifies NameNode of what blocks it has

• NameNode replicates blocks 2x in local rack, 1x elsewhere

Hadoop’s Architecture: MapReduce Engine
Hadoop’s Architecture
MapReduce Engine:

• JobTracker & TaskTracker

• JobTracker splits up data into smaller tasks(“Map”) and sends

it to the TaskTracker process in each node

• TaskTracker reports back to the JobTracker node and reports

on job progress, sends data (“Reduce”) or requests new jobs
Hadoop’s Architecture
• None of these components are necessarily limited to using
HDFS

• Many other distributed file-systems with quite different

architectures work

• Many other software packages besides Hadoop's

MapReduce platform make use of HDFS
Hadoop in the Wild
• Hadoop is in use at most organizations that handle big data:
o Yahoo!
o Facebook
o Amazon
o Netflix
o Etc…

• Some examples of scale:

o Yahoo!’s Search Webmap runs on 10,000 core Linux
cluster and powers Yahoo! Web search

o FB’s Hadoop cluster hosts 100+ PB of data (July, 2012) &

growing at ½ PB/day (Nov, 2012)
Hadoop in the Wild
Three main applications of Hadoop:

• Advertisement (Mining user behavior to generate

recommendations)

• Searches (group related documents)

• Security (search for uncommon patterns)

Hadoop in the Wild
• Non-realtime large dataset computing:

o NY Times was dynamically generating PDFs of articles

from 1851-1922

o Wanted to pre-generate & statically serve articles to

improve performance

o Using Hadoop + MapReduce running on EC2 / S3,

converted 4TB of TIFFs into 11 million PDF articles in 24
hrs
Hadoop in the Wild: Facebook Messages

• Design requirements:

o Integrate display of email, SMS and

chat messages between pairs and
groups of users

o Strong control over who users

receive messages from

o Suited for production use between

500 million people immediately after
launch

o Stringent latency & uptime

requirements
 • System requirements
Hadoop in the Wild
o High write throughput

o Cheap, elastic storage

o Low latency

o High consistency (within a

single data center good
enough)

o Disk-efficient sequential and

random read performance
Hadoop in the Wild

• Classic alternatives

o These requirements typically met using large MySQL cluster &

caching tiers using Memcached

o Content on HDFS could be loaded into MySQL or Memcached

if needed by web tier

• Problems with previous solutions

o MySQL has low random write throughput… BIG problem for

messaging!

o Difficult to scale MySQL clusters rapidly while maintaining

performance

o MySQL clusters have high management overhead, require

more expensive hardware
Hadoop in the Wild
• Facebook’s solution

o Hadoop + HBase as foundations

o Improve & adapt HDFS and HBase to scale to FB’s workload

and operational considerations

 Major concern was availability: NameNode is SPOF &

failover times are at least 20 minutes

 Proprietary “AvatarNode”: eliminates SPOF, makes HDFS

safe to deploy even with 24/7 uptime requirement

 Performance improvements for realtime workload: RPC

timeout. Rather fail fast and try a different DataNode
Hadoop Highlights
Distributed File System
Fault Tolerance
Open Data Format
Flexible Schema
Queryable Database
Why use Hadoop?
Need to process Multi Petabyte Datasets
Data may not have strict schema
Expensive to build reliability in each application
Nodes fails everyday
Need common infrastructure
Very Large Distributed File System
Assumes Commodity Hardware
Optimized for Batch Processing
Runs on heterogeneous OS
DataNode

A Block Sever
 Stores data in local file system
 Stores meta-data of a block - checksum
 Serves data and meta-data to clients
Block Report
 Periodically sends a report of all existing blocks to
NameNode
Facilitate Pipelining of Data
 Forwards data to other specified DataNodes
Block Placement
Replication Strategy
One replica on local node
Second replica on a remote rack
Third replica on same remote rack
Additional replicas are randomly placed
Clients read from nearest replica
Data Correctness
Use Checksums to validate data – CRC32
File Creation
Client computes checksum per 512 byte
DataNode stores the checksum
File Access
Client retrieves the data and checksum from DataNode
If validation fails, client tries other replicas
Data Pipelining
Client retrieves a list of DataNodes on which to place replicas of a
block
Client writes block to the first DataNode
The first DataNode forwards the data to the next DataNode in the
Pipeline
When all replicas are written, the client moves on to write the next
block in file
Goals of HDFS
Very Large Distributed File System
10K nodes, 100 million files, 10PB
Assumes Commodity Hardware
Files are replicated to handle hardware failure
Detect failures and recover from them
Optimized for Batch Processing
Data locations exposed so that computations
can move to where data resides
Provides very high aggregate bandwidth
Distributed File System

Single Namespace for entire cluster

Data Coherency
 Write-once-read-many access model
 Client can only append to existing files
Files are broken up into blocks
 Typically 64MB block size
 Each block replicated on multiple DataNodes
Intelligent Client
 Client can find location of blocks
 Client accesses data directly from DataNode
HDFS Architecture
NameNode Metadata

Metadata in Memory
 The entire metadata is in main memory
 No demand paging of metadata
Types of metadata
 List of files
 List of Blocks for each file
 List of DataNodes for each block
 File attributes, e.g. creation time, replication factor
A Transaction Log
 Records file creations, file deletions etc
DataNode

A Block Server
Stores data in the local file system (e.g. ext3)
Stores metadata of a block (e.g. CRC)
Serves data and metadata to Clients
Block Report
Periodically sends a report of all existing blocks
to the NameNode
Facilitates Pipelining of Data
Forwards data to other specified DataNodes
Block Placement

Current Strategy
One replica on local node
Second replica on a remote rack
Third replica on same remote rack
Additional replicas are randomly placed
Clients read from nearest replicas
Would like to make this policy pluggable
Heartbeats

DataNodes send hearbeat to the NameNode

Once every 3 seconds
NameNode uses heartbeats to detect
DataNode failure
Replication Engine
NameNode detects DataNode failures
Chooses new DataNodes for new replicas
Balances disk usage
Balances communication traffic to DataNodes
Data Correctness
Use Checksums to validate data
Use CRC32
File Creation
Client computes checksum per 512 bytes
DataNode stores the checksum
File access
Client retrieves the data and checksum from
DataNode
If Validation fails, Client tries other replicas
Data Pieplining

Client retrieves a list of DataNodes on which

to place replicas of a block
Client writes block to the first DataNode
The first DataNode forwards the data to the
next node in the Pipeline
When all replicas are written, the Client
moves on to write the next block in file
Rebalancer
Goal: % disk full on DataNodes should be
similar
Usually run when new DataNodes are added
Cluster is online when Rebalancer is active
Rebalancer is throttled to avoid network
congestion
Command line tool
Secondary NameNode
Copies FsImage and Transaction Log from
Namenode to a temporary directory
Merges FSImage and Transaction Log into a
new FSImage in temporary directory
Uploads new FSImage to the NameNode
Transaction Log on NameNode is purged
 Commads for HDFS User:
User Interface
hadoop dfs -mkdir /foodir
hadoop dfs -cat /foodir/myfile.txt
hadoop dfs -rm /foodir/myfile.txt
Commands for HDFS Administrator
hadoop dfsadmin -report
hadoop dfsadmin -decommision datanodename
Web Interface
http://host:port/dfshealth.jsp