Hadoop is an open source framework for distributed storage and processing of large datasets across clusters of computers. It was created in 2005 to address limitations of earlier systems for large scale computing like moving large amounts of data. Hadoop uses a distributed file system to store data across nodes and a programming model called MapReduce to distribute computation tasks.
Hadoop is an open source framework for distributed storage and processing of large datasets across clusters of computers. It was created in 2005 to address limitations of earlier systems for large scale computing like moving large amounts of data. Hadoop uses a distributed file system to store data across nodes and a programming model called MapReduce to distribute computation tasks.
Hadoop is an open source framework for distributed storage and processing of large datasets across clusters of computers. It was created in 2005 to address limitations of earlier systems for large scale computing like moving large amounts of data. Hadoop uses a distributed file system to store data across nodes and a programming model called MapReduce to distribute computation tasks.
Hadoop is an open source framework for distributed storage and processing of large datasets across clusters of computers. It was created in 2005 to address limitations of earlier systems for large scale computing like moving large amounts of data. Hadoop uses a distributed file system to store data across nodes and a programming model called MapReduce to distribute computation tasks.
for storage and processing of large scale data on clusters of commodity hardware
Created by Doug Cutting and Mike Carafella
in 2005.
Cutting named the program after his son’s
toy elephant. Data-intensive text processing
Assembly of large genomes
Graph mining
Machine learning and data mining
Large scale social network analysis
Hadoop Common •Contains Libraries and other modules
HDFS •Hadoop Distributed File System
Hadoop YARN •Yet Another Resource Negotiator
Hadoop •A programming model for large scale
MapReduce data processing What considerations led to its design What were the limitations of earlier large- scale computing?
What requirements should an alternative
approach have?
How does Hadoop address those
requirements? Historically computation was processor- bound ◦ Data volume has been relatively small ◦ Complicated computations are performed on that data Advances in computer technology has historically centered around improving the power of a single machine Allows developers to use multiple machines for a single task Multiple computing resources are connected Computing task is distributed across resources Deals with tables & relations Must have schema for data Implements data fragmentation and partitioning Generate notations for transactions Allow distributed transactions Suitable to process huge data in limited time Programming on a distributed system is much more complex ◦ Synchronizing data exchanges ◦ Managing a finite bandwidth ◦ Controlling computation timing is complicated
Distributed systems must be designed with the
expectation of failure Typically divided into Data Nodes and Compute Nodes At compute time, data is copied to the Compute Nodes Fine for relatively small amounts of data
Modern systems deal with far more data than
was gathering in the past Facebook ◦ 500 TB per day Yahoo ◦ Over 170 PB (Petabyte=1000TB) eBay ◦ Over 6 PB
Getting the data to the processors becomes
the bottleneck Must support partial failure
◦ If a component fails, it should be able to recover without
restarting the entire system
◦ Component failure or recovery during a job must not
affect the final output
Must be scalable ◦ Increasing resources should increase load capacity
◦ Increasing the load on the system should result in a
graceful decline in performance for all jobs ◦ Not system failure Based on work done by Google in the early 2000s ◦ “The Google File System” in 2003 ◦ “MapReduce: Simplified Data Processing on Large Clusters” in 2004 The core idea was to distribute the data as it is initially stored ◦ Each node can then perform computation on the data it stores without moving the data for the initial processing Applications are written in a high-level programming language ◦ No network programming or temporal dependency Nodes should communicate as little as possible ◦ A “shared nothing” architecture Data is spread among the machines in advance ◦ Perform computation where the data is already stored as often as possible When data is loaded onto the system it is divided into blocks ◦ Typically 64MB or 128MB Tasks are divided into two phases ◦ Map tasks which are done on small portions of data where the data is stored ◦ Reduce tasks which combine data to produce the final output A master program allocates work to individual nodes HDFS Responsible for storing data on the cluster
Data files are split into blocks and distributed
across the nodes in the cluster
Each block is replicated multiple times
HDFS is a file system written in Java based on the Google’s GFS
Provides redundant storage for massive
amounts of data HDFS works best with a smaller number of large files ◦ Millions as opposed to billions of files ◦ Typically 100MB or more per file
Files in HDFS are write once
Optimized for streaming reads of large files
and not random reads Files are split into blocks Blocks are split across many machines at load time ◦ Different blocks from the same file will be stored on different machines Blocks are replicated across multiple machines The NameNode keeps track of which blocks make up a file and where they are stored Default replication is 3-fold Secondary namenode takes snapshot of primary namenode after regular interval Name node is for storage and management of metadata JobTracker assign task to task tracker Each DataNode send “”Heart Beat Signal” to NameNode for active/inactive status When a client wants to retrieve data
◦ Communicates with the NameNode to determine
which blocks make up a file and on which data nodes those blocks are stored
◦ Then communicated directly with the data nodes to
read the data Distributing computation across nodes A method for distributing computation across multiple nodes
Each node processes the data that is stored at
that node
Consists of two main phases
◦ Map ◦ Reduce Reads data as key/value pairs ◦ The key is often discarded
Outputs zero or more key/value pairs
Output from the mapper is sorted by key
All values with the same key are guaranteed
to go to the same machine Called once for each unique key
Gets a list of all values associated with a key
as input
The reducer outputs zero or more final
key/value pairs ◦ Usually just one output per input key What parts actually make up a Hadoop cluster NameNode ◦ Holds the metadata for the HDFS Secondary NameNode ◦ Performs housekeeping functions for the NameNode DataNode Stores the actual HDFS data blocks Secondary Name Node ◦ Hourly Back up ◦ This data will put in the fs_image if the cluster fails than this file loads on any other system and revoke our hadoop Stores the HDFS file system information in a fsimage
Updates to the file system (add/remove
blocks) do not change the fsimage file ◦ They are instead written to a log file
When starting the NameNode loads the
fsimage file and then applies the changes in the log file NOT a backup for the NameNode
Periodically reads the log file and applies the
changes to the fsimage file bringing it up to date
Allows the NameNode to restart faster when
required JobTracker ◦ Determines the execution plan for the job ◦ Assigns individual tasks
TaskTracker ◦ Keeps track of the performance of an individual mapper or reducer Other available tools MapReduce is very powerful, but can be awkward to master
These tools allow programmers who are
familiar with other programming styles to take advantage of the power of MapReduce Hive ◦ Hadoop processing with SQL Pig ◦ Hadoop processing with scripting
