The solution for Big data
HADOOP
J. Sai Krishna
and G. Sravya
Lahari
2nd B.Tech (CSE)
K.O.R.M College of Engineering
Kadapa
Contents
1. Data trends in storing data.
2. Bigdata problems in IT industry
3. Introduction to HADOOP
4. HDFS (Hadoop Distributed File System)
MapReduce
6. Prominent users of Hadoop.
7. Conclusion
5.
Data trends in storing data
What is data--- Any real world symbol
(character,
numeric, special character)
or a of group of
them is
said to be
data it may be of the visual or audio or
scriptural ,etc
Big data
What is big dataIn IT, it is a collection
of data sets so large and complex data that
it becomes difficult to process using onhand database management tools or
traditional data processing applications.
As of 2012, limits on the size of data sets
that are feasible to process in reasonable
time were on the order of Exabyte of data.
BIGDATA and problems with
it.
Daily about 0.5 Petabytes of updates are being made
into FACEBOOK including 40 millions photos.
Daily, YOUTUBE is loaded with videos that can be
watched for one year continuously
Limitations are encountered due to large data sets in
many areas, including meteorology, genomics,
complex physics simulations, and biological and
environmental research.
Also affect Internet search, finance and business
informatics.
The challenges include in capture, retrieval, storage,
search, sharing, analysis, and visualization.
THEN WHAT COULD BE THE
SOLUTION FOR BIGDATA
HADOOP
What is Hadoop?
It is a opensource software written in java
Hadoop software library is a framework that
allows for the distributed processing of
large data sets across clusters of
computers using simple programming
models.
It is designed to scale up from single
servers to thousands of machines, each
offering local computation and storage.
The project includes these
modules:
Hadoop Common
Hadoop Distributed File System
(HDFS)
Hadoop MapReduce
1.Hadoop Commons
It provides access to the filesystems
supported by Hadoop.
The Hadoop Common package contains the
necessary JAR files and scripts needed to
start Hadoop.
The package also provides source code,
documentation, and a contribution section
which includes projects from the Hadoop
Community (Avro, Cassandra, Chukwa,
Hbase, Hive, Mahout, Pig, ZooKeeper)
2. Hadoop Distributed File
System (HDFS):
Hadoop uses HDFS, a distributed file
system based on GFS (Google File System),
as its shared filesystem.
HDFS architecture divides files into large
chunks (~64MB) distributed across data
servers (this is configurable).
It has a namenode and datanodes
What does a HDFS contain
HDFS consists of a global namenodes or
namespaces and they are federated.
The datanodes are used as common
storage for blocks by all the Namenodes.
Each datanode registers with all the
Namenodes in the cluster.
Datanodes send periodic heartbeats and
block reports and handles commands
from the Namenodes
Structure of Hadoop system:
MASTER NODE
Master node
Keeps
track of namespace and metadata about items
Keeps track of MapReduce jobs in the system
Hadoop currently configured with centurion064 as
the master node
Hadoop is locally installed in each system.
Installed location is in /localtmp/hadoop/hadoop0.15.3
SLAVE NODES
Slave nodes
Manage
blocks of data sent from master node
In common, these are the chunkservers
Currently centurion060, centurion064 are the two
slave nodes being used.
Slave nodes store their data in
/localtmp/hadoop/hadoop-dfs (this is automatically
created by the DFS)
Once you use the DFS, relative paths are from
/usr/{your usr id}
Advantages and Limitations of
HDFS
Reduce traffic on job
scheduling.
File access can be
achieved through
the native Java or
language of the
users' choice (C++,
Java, Python, PHP,
Ruby, Erlang, Perl,
Haskell, C#, Cocoa,
Smalltalk, and
OCaml),
It cannot be
directly mounted
by an existing
operating system.
It should be
provided with UNIX
or LUNIX system.
3.Hadoop MAPREDUCE
SYSTEM
MAP AND REDUCE METHODS USAGE
Map function
Reduce function
Run this program as a
MapReduce job
WORD COUNT OVER A GIVEN
SET OF STRINGS
We love India
We play
tennis
We
1
love
1
India
We
1
Play
Map
1
tennis
Love
India
1
1
We
2
tennis 1
play
1
Reduce
MAPREDUCE IN WITH NO REDUCE TASKS
MAPREDUCE WITH TWO REDUCE
TASKS - AUTOMATIC PARALLEL
EXECUTION IN MAPREDUCE
MapReduce - lifecycle
Input
Splits
Map
function
Map phase
Reduce
function
Reduce phase
Shuffle and sort in MapReduce
with multiple reduce tasks
Prominent users of HADOOP
Amazon 100 nodes
Facebook two clusters of 8000 and 3000
nodes
Adobe 80 node system
EBay 532 node cluster
yahoo cluster of about 4500 nodes
IIIT Hyderabad 30 node cluster
Achievements
March 2011 - Apache Hadoop takes top
prize at Media Guardian Innovation Award
July 2012 - Hadoop Wins Terabyte Sort
Benchmark
Conclusion:
It reduce traffic on capture, storage, search,
sharing, analysis, and visualization.
A huge amount of data could be stored and large
computations could be done in a single
compound with full safety and security at cheap
cost.
BIGDATA and BIGDATA-SOLUTIONS is one of the
burning issues in the present IT industry so, work
on those will surely make you more useful to that.
Thank
you
Any queries
