Scribd
Upload
194 views30 pages

Hadoop Distributed File System Basics

Hadoop Distributed File System (HDFS) is designed for distributed storage and processing of large datasets across clusters of computers. HDFS uses a master/slave architecture with a NameNode that manages metadata and DataNodes that store data blocks. Data is replicated across multiple DataNodes for fault tolerance. The MapReduce programming model is used for distributed processing of data in HDFS through parallel mapping and reducing of data blocks.

Uploaded by

ashuvasuma
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
194 views30 pages

Hadoop Distributed File System Basics

Hadoop Distributed File System (HDFS) is designed for distributed storage and processing of large datasets across clusters of computers. HDFS uses a master/slave architecture with a NameNode that manages metadata and DataNodes that store data blocks. Data is replicated across multiple DataNodes for fault tolerance. The MapReduce programming model is used for distributed processing of data in HDFS through parallel mapping and reducing of data blocks.

Uploaded by

ashuvasuma
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 30

Hadoop Distributed File

System Basics
Contents
• Hadoop Distributed File System Design Features
• HDFS Components
• HDFS Block Replication
• HDFS Safe Mode
• Rack Awareness
• NameNode High Availability
• HDFS Checkpoints and Backups
• HDFS Snapshots
• HDFS NFS Gateway
Hadoop Distributed File System
Design Features
• Big data processing
• write-once/read-many model
• No caching of data
• Design based on Google File System(GFS)
• Designed for Data Streaming
• Data Locality
• Moving Computation
HDFS Components
• Node Types
• Name Node - Manage Metadata
• Data Node – Store/Retrieve Data

•Design-Master/Slave Architecture
• Master(NN)-File System Namespace
• Slave(DN)-Read/Write Request
Various system roles in an HDFS deployment
• Disk Files-
• fsimage_*
• image of the file system state used only at startup by the NameNode.
• Stores metadata

• edit_*
• A series of modifications done to the file system after starting the NameNode.

• Location -
dfs.namenode.name.dir property in the hdfs-site.xml file.
HDFS Block Replication Cluster Replication
Factor
>8 3
• Replicates across the cluster >1 and <8 2

• amount of replication - dfs.replication Single


Machine
1

in hdfs-site.xml file
• Replicating each block across number of
machine(default=3)
• HDFS based on block size default is
64MB
• Splits are based on logical partitioning
of data.
HDFS Safe Mode
• When the NameNode starts
• enters a read-only safe mode
• blocks cannot be replicated or deleted
• Safe Mode enables the NameNode to perform two important
processes:
• Loading fsimage file into memory and replaying the edit log
• Mapping between blocks and data nodes is created,at least one copy of the
data is available before safe mode exit.
• Safe mode for maintenance - hdfs dfsadmin-safemode command
-Administrator maintenance
Rack Awareness
• Data Locality
• Hadoop MapReduce is to move the computation to the data
1. Data resides on the local machine (best).
2. Data resides in the same rack (better).
3. Data resides in a different rack (good).
Example- YARN Scheduler
Pros- improved fault tolerance
Cons-Entire rack failure then performance degraded
NameNode High Availability
• Earlier,NameNode was a single point of failure
• NameNode High Availability (HA) -to provide
true failover service
HA

Active Standby
Name node Name node

all client HDFS maintains enough state to


operations in the provide a fast failover
cluster

Apache ZooKeeper is used to monitor the NameNode


health.
HDFS NameNode Federation
• Earlier,single
namespace/nameNode
• Federation addresses this
limitation
• Multiple nameNodes/namespaces
• Key benefits:
• Namespace scalability
• Better performance
• System isolation
HDFS Checkpoints and Backups
• Checkpoints:
• fsimage - stores metedata
• edit log file - file system
modifications
• Backups:
• Maintains an up-to-date copy of the file system namespace both in
memory and on disk
• NameNode supports one BackupNode at a time
• No CheckpointNodes may be registered if a Backup node is in use.
HDFS Snapshots
• Created by administrators using the hdfs dfs snapshot command.
• HDFS snapshots are read-only point-in-time copies of the file system
• Features:
• sub-tree of the file system or the entire file system
• data backup, protection against user errors, and disaster recovery
• Instant creation
• Data not copied
• record the block list and the file size
• Doesnot affect regular HDFS operations.
Hadoop MapReduce
Framework
Contents
• MapReduce Model
• MapReduce Parallel Data Flow
• Data flow for a word count program
• Process placement during MapReduce
• Fault Tolerance
• Speculative Execution
MapReduce Model
• Simple, powerful
• Stages:
• Mapping stage
• Reducing stage
• Example word count
• grep " <<word>>" <<filename>> | wc -l/c
• The mapper and reducer functions are both defined with respect to
data structured in (key, value) pairs.
• The mapper takes one pair of data with a type in one data domain,
and returns a list of pairs in a different domain:
• Map(key1,value1) → list(key2,value2)
• The reducer function is then applied to each key–value pair, which in
turn produces a collection of values in the same domain:
• Reduce(key2, list (value2)) → list(value3)
• Each reducer call typically produces either one value (value3) or an
empty response. Thus, the MapReduce framework transforms a list of
(key, value) pairs into a list of values.
• Properties:
• Data flow is in one direction (map to reduce)
• Original data is preserverd
• No dependency
• Hadoop accomplishes parallelism by using a distributed file system
(HDFS) to slice and spread data over multiple servers.
• Data slice are then combined in the reducer step
MapReduce Parallel Data Flow
• Input Splits-HDFS distributes and replicates data over multiple servers
• Map Step-Parallel nature of Hadoop
• Combiner Step-Key–value pairs are combined prior
• Shuffle Step-similar keys are combined and counted by reducer
• Reduce Step-Actual Reduction happens and outputs to HDFS
Simple Hadoop MapReduce data flow for a word
count program
The count for run can be
combined into (run,2) before
the shuffle. This optimization
can help minimize the
amount of data transfer
needed for the shuffle phase.
• Placement of mappers and reducers are done by Hadoop YARN
resource manager and MapReduce framework.
• Nodes can run both mapper and reducer tasks.
• The dynamic nature of YARN enables the work containers used by
completed map tasks to be returned to the pool of available
resources.
• The shuffle stage makes sure the necessary data are sent to each
mapper.
• Mappers can complete at any time.
Process placement during MapReduce
Fault Tolerance
• Strict control of data flow throughout the execution of the program.
• Design of MapReduce makes it possible to easily recover from the
failure of one or many map processes as it has no dependency.
• Failed reducers can be restarted.
• MapReduce ApplicationMaster will need to restart the reducer tasks
• Process is totally transparent to the user and provides a fault-tolerant
system to run applications.
Speculative Execution
• Challenges with large clusters-inability to predict or manage
unexpected system failures.
• Congested network, slow disk controller, failing disk, high processor
load, or similar problem lead to slow performance.
• One part of a MapReduce process runs slowly, then the application
cannot complete.
• As input data are immutable in the MapReduce process, start a copy
of a running map process without disturbing any other running
mapper processes.
• Speculative execution can reduce cluster efficiency,it can be turned
off and on in the mapred-site.xml configuration file.
Node A Task
Node 1
Progress
Task(Slow)

Scheduler
Node 2
Node B Launch
Speculative
Task(Duplicate)

Node 3

You might also like