Scribd
Upload
23 views

Big Data Unit 1 Notes

Uploaded by

jyotsnas99
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
23 views

Big Data Unit 1 Notes

Uploaded by

jyotsnas99
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 37

SWAMI VIVEKANANDA INSTITUTE OF

TECHNOLOGY
TOPICS

UNIT - I Introduction to Big Data: Big Data and its Importance – Four V’s of
Big Data – Drivers for Big Data – Introduction to Big Data Analytics – Big Data
Analytics applications.
Introduction to Big Data:

What is Big Data?


Big Data is a collection of data that is huge in volume, yet growing exponentially
with time.
It is a data with so large size and complexity that none of traditional data
management tools can store it or process it efficiently. Big data is also a data but
with huge size.

What is an Example of Big Data?


Following are some of the Big Data examples-

The New York Stock Exchange is an example of Big Data that generates about one
terabyte of new trade data per day.

Social Media

The statistic shows that 500+terabytes of new data get ingested into the databases
of social media site Facebook, every day. This data is mainly generated in terms of
photo and video uploads, message exchanges, putting comments etc.

A single Jet engine can generate 10+terabytes of data in 30 minutes of flight time.
With many thousand flights per day, generation of data reaches up to
many Petabytes.

Types of data:

The digital data is divided into three types

1.unstructured data: This is the data which does not conform to a data model or
is not in a form which can be used easily by a computer program.
Forexample:

chatrooms,memos,power point presentation,images,videos,letters,researches,white


papers,body of an email.

Semi-structured data:

This is the data which does not conform to a data model but has some
structure.how ever it is not in the form which can be used easily by a computer
program.

Examples:E-mails,XML,markup languages,HTML

Structured data: This is the data which is in the organized form. i.e in the form of
rows and columns and can be easily used by a computer programs. Relationships
exist between the entities of data, such as classes and their objects.Data stored in
data bases is an example of structured data.

Structured data

Digital Semi structured


Data

unstructured

Structured data:
The data when it conforms to the schema/structure we say it is structured data.

 Structured data is generally tabular data that is represented by columns and rows in a
database.

 Databases that hold tables in this form are called relational databases.

 The mathematical term “relation” specify to a formed set of data held as a table.

 In structured data, all row in a table has the same set of columns.

 SQL (Structured Query Language) programming language used for structured data.
Sources of structured data:
The sources from where the data is generated is RDBMS,oracle,DB2,Microsoft
Sql server,Teradata,Mysql and OLTP systems.
Semi structured data:
The semi structured data is also referred to as self describing tags.
It uses tags to segregate the semantic elements.
Sources of semi structured data:
The sources of semi structured are
XML-Extensible mark up language
JSON-Java script object Notation.
An example of HTML as follows
<html>
<head>
<title>place your title here
</head>
<body bgcolor=”FFFFFF”
</html>

Sample JSON document


{
_id:9
Booktitle:Fundamentals of Business Analyticts’
Author Name:”Seema acharya”
Publisher:”Wiley India”
Year of Publication:”2011”
}
Unstructured data:
Unstructured data does not conform to any predefined model.
Sources of Unstructured data:
How to deal with unstructured data:
Characteristics of Data:
1.Composition: The composition of data deals with the structure of data, that
is, the sources of data, the granularity, the types, and the nature of data as to
whether it is static or real-time streaming.
2.Condition: The condition of data deals with the state of data, that is, "Can
one use this data as is for analysis?" or "Does it require cleansing for further
enhancement and enrichment?"
3.Context: The context of data deals with "Where has this data been
generated?" "Why was this data generated?" How sensitive is this data?"
"What are the events associated with this data?" and so on.
Small data (data as it existed prior to the big data revolution) is about certainty.
It is about known data sources; it is about no major changes to the composition
or context of data.

Composition

Data Condition

Context

Evolution of Big Data:

1970s and before was the era of mainframes. The data was
essentially primitive and structured. Relational databases
evolved in 1980s and 1990s. The era was of data intensive
applications. The World Wide Web (WWW) and the Internet of
Things (IOT) have led to an onslaught of structured,
unstructured, and multimedia data. Refer Table 1.1.

Big Data and its Importance

The importance of big data does not revolve around how much data a company
has but how a company utilizes the collected data. Every company uses data in
its own way; the more efficiently a company uses its data, the more potential it
has to grow. The company can take data from any source and analyze it to find
answers which will enable:

Big Data importance doesn’t revolve around the amount of data a company
has but lies in the fact that how the company utilizes the gathered data.
Every company uses its collected data in its own way. More effectively the
company uses its data, more rapidly it grows.
By analysing the big data pools effectively the companies can get answers to :
Cost Savings :
o Some tools of Big Data like Hadoop can bring cost advantages to business
when large amounts of data are to be stored.
o These tools help in identifying more efficient ways of doing business.
Time Reductions :
o The high speed of tools like Hadoop and in-memory analytics can easily
identify new sources of data which helps businesses analyzing data
immediately.
o This helps us to make quick decisions based on the learnings.
Understand the market conditions :
o By analyzing big data we can get a better understanding of current market
conditions.
o For example: By analyzing customers’ purchasing behaviours, a company
can find out the products that are sold the most and produce products
according to this trend. By this, it can get ahead of its competitors.
Control online reputation :
o Big data tools can do sentiment analysis.
o Therefore, you can get feedback about who is saying what about your
company.
o If you want to monitor and improve the online presence of your business,
then big data tools can help in all this.
Using Big Data Analytics to Boost Customer Acquisition(purchase) and
Retention :
o The customer is the most important asset any business depends on.
o No single business can claim success without first having to establish a solid
customer base.
o If a business is slow to learn what customers are looking for, then it is very
likely to deliver poor quality products.
o The use of big data allows businesses to observe various customer-related
patterns and trends.
Using Big Data Analytics to Solve Advertisers Problem and Offer
Marketing Insights :
o Big data analytics can help change all business operations.
o Like the ability to match customer expectations, changing
company’s product line, etc.
o And ensuring that the marketing campaigns are powerful

Four v’s Of Big Data


In recent years, Big Data was defined by the “3Vs” but now there is “6Vs” of
Big Data which are also termed as the characteristics of Big Data as follows:

1. Volume:
 The name ‘Big Data’ itself is related to a size which is enormous.
 Volume is a huge amount of data.
 To determine the value of data, size of data plays a very crucial
role. If the volume of data is very large then it is actually considered
as a ‘Big Data’. This means whether a particular data can actually
be considered as a Big Data or not, is dependent upon the volume
of data.
 Hence while dealing with Big Data it is necessary to consider a
characteristic ‘Volume’.
 Example: In the year 2016, the estimated global mobile traffic was
6.2 Exabytes(6.2 billion GB) per month. Also, by the year 2020 we
will have almost 40000 Exa Bytes of data.
2. Velocity:

 Velocity refers to the high speed of accumulation of data.


 In Big Data velocity data flows in from sources like machines,
networks, social media, mobile phones etc.
 There is a massive and continuous flow of data. This determines
the potential of data that how fast the data is generated and
processed to meet the demands.
 Sampling data can help in dealing with the issue like ‘velocity’.
 Example: There are more than 3.5 billion searches per day are
made on Google. Also, FaceBook users are increasing by 22%
(Approx.) year by year.
3. Variety:
 It refers to nature of data that is structured, semi-structured and
unstructured data.
 It also refers to heterogeneous sources.
 Variety is basically the arrival of data from new sources that are
both inside and outside of an enterprise. It can be structured, semi-
structured and unstructured.
 Structured data: This data is basically an organized data.
It generally refers to data that has defined the length and
format of data.
 Semi- Structured data: This data is basically a semi-
organised data. It is generally a form of data that do not
conform to the formal structure of data. Log files are the
examples of this type of data.
 Unstructured data: This data basically refers to
unorganized data. It generally refers to data that doesn’t fit
neatly into the traditional row and column structure of the
relational database. Texts, pictures, videos etc. are the
examples of unstructured data which can’t be stored in the
form of rows and columns.
4. Veracity:
 It refers to inconsistencies and uncertainty in data, that is data
which is available can sometimes get messy and quality and
accuracy are difficult to control.
 Big Data is also variable because of the multitude of data
dimensions resulting from multiple disparate data types and
sources.
 Example: Data in bulk could create confusion whereas less amount
of data could convey half or Incomplete Information.
5. Value:
 After having the 4 V’s into account there comes one more V which
stands for Value!. The bulk of Data having no Value is of no good to
the company, unless you turn it into something useful.
 Data in itself is of no use or importance but it needs to be converted
into something valuable to extract Information. Hence, you can
state that Value! is the most important V of all the 6V’s.
6. Variability:
 How fast or, available data that extent is the structure of your data
is changing ?.
 How often does the meaning or shape of your data!to ?change?.
 example : if you are eating same ice-cream daily and the taste just
keep changing.
Drivers for Big Data
Big Data has quickly risen to become one of the most desired topics in the
industry.
The main business drivers for such rising demand for Big Data Analytics are :
1. The digitization of society
2. The drop in technology costs
3. Connectivity through cloud computing
4. Increased knowledge about data science
5. Social media applications
6. The rise of Internet-of-Things(IoT)
Example: A number of companies that have Big Data at the core of their
strategy like :
Apple, Amazon, Facebook and Netflix have become very successful at the
beginning of the 21st century.

1. The digitization of society


Big Data is largely consumer driven and consumer oriented. Most of the data
in the world is generated by consumers, who are nowadays ‘always-on’.
Most people now spend 4-6 hours per day consuming and generating data
through a variety of devices and (social) applications.
With every click, swipe or message, new data is created in a database
somewhere around the world.
Because everyone now has a smartphone in their pocket, the data creation
sums to incomprehensible amounts.
Some studies estimate that 60% of data was generated within the last two
years, which is a good indication of the rate with which society has digitized.

2. The drop in technology costs


Technology related to collecting and processing massive quantities of diverse (high variety)
data has become increasingly more affordable.
The costs of data storage and processors keep declining, making it possible for small
businesses and individuals to become involved with Big Data.
For storage capacity, the often-cited Moore’s Law still holds that the storage density (and
therefore capacity) still doubles every two years.
The plummeting of technology costs has been depicted in the figure below.

Besides the plummeting of the storage costs, a second key contributing factor to the
affordability of Big Data has been the development of open source Big Data software
frameworks.

The most popular software framework (nowadays considered the standard for Big Data) is
Apache Hadoop for distributed storage and processing.

Due to the high availability of these software frameworks in open sources, it has become
increasingly inexpensive to start Big Data projects in organizations.

3. Connectivity through cloud computing


Cloud computing environments (where data is remotely stored in distributed storage systems)
have made it possible to quickly scale up or scale down IT infrastructure and facilitate a pay-
as-you-go model.

This means that organizations that want to process massive quantities of data (and thus have
large storage and processing requirements) do not have to invest in large quantities of IT
infrastructure.
Instead, they can license the storage and processing capacity they need and only pay for the
amounts they actually used. As a result, most of Big Data solutions leverage the possibilities of
cloud computing to deliver their solutions to enterprises.

4. Increased knowledge about data science


In the last decade, the term data science and data scientist have become tremendously
popular. In October 2012, Harvard Business Review called the data scientist “sexiest job of
the 21st century” and many other publications have featured this new job role in recent years.
The demand for data scientist (and similar job titles) has increased tremendously and many
people have actively become engaged in the domain of data science.

As a result, the knowledge and education about data science has greatly professionalized and
more information becomes available every day. While statistics and data analysis mostly
remained an academic field previously, it is quickly becoming a popular subject among
students and the working population.

5. Social media applications


Everyone understands the impact that social media has on daily life. However, in the
study of Big Data, social media plays a role of paramount importance. Not only because of
the sheer volume of data that is produced everyday through platforms such as Twitter,
Facebook, LinkedIn and Instagram, but also because social media provides nearly real-time
data about human behavior.
Social media data provides insights into the behaviors, preferences and opinions of ‘the
public’ on a scale that has never been known before. Due to this, it is immensely valuable to
anyone who is able to derive meaning from these large quantities of data. Social media data
can be used to identify customer preferences for product development, target new customers
for future purchases, or even target potential voters in elections. Social media data might even
be considered one of the most important business drivers of Big Data.

6. The upcoming internet of things (IoT)


The Internet of things (IoT) is the network of physical devices, vehicles, home appliances and
other items embedded with electronics, software, sensors, actuators, and network connectivity
which enables these objects to connect and exchange data.

It is increasingly gaining popularity as consumer goods providers start including ‘smart’ sensors
in household appliances. Whereas the average household in 2010 had around 10 devices that
connected to the internet, this number is expected to rise to 50 per household by 2020.

Examples of these devices include thermostats, smoke detectors, televisions, audio systems and
even smart refrigerators.
● Medical information, such as diagnostic imaging
● Photos and video footage uploaded to the World Wide Web
● Video surveillance, such as the thousands of video cameras
across a city
● Mobile devices, which provide geospatial location data of
the
users
● Metadata about text messages, phone calls, and application
usage on smart phones
● Smart devices, which provide sensor-based collection of
information from smart
● Non traditional IT devices, including the use of RFID
readers,
GPS navigation systems, and seismic processing.
These are the multiple sources where the data can be
generated from multiple sources.

Challenges of Big Data


The major challenges associated with big data are as follows −

 Capturing data
 Curation
 Storage
 Searching
 Sharing
 Transfer
 Analysis
 Presentation
1.Data is growing in an exponential rate.Most of the data have been generated in the last 2-3
years.

Introduction to Big Data Analyticts

Organizations and data collectors are realizing that the data they can gather
from individuals contains intrinsic value and, as a result, a new economy is
emerging.

As this new digital economy continues to evolve, the market sees the
introduction of data vendors and data cleaners that use crowdsourcing to test the
outcomes of machine learning techniques.

Other vendors offer added value by repackaging open source tools in a simpler
way and bringing the tools to market. Vendors such as Cloudera, Hortonworks,
and Pivotal have provided this value-add for the open source framework
Hadoop.
Data devices and the “Sensornet” gather data from multiple locations and
continuously generate new data about this data. For each gigabyte of new data
created, an additional petabyte of data is created about that data.

For example, consider someone playing an online video game through a PC,
game console, or smartphone. In this case, the video game provider captures
data about the skill and levels attained by the player. Intelligent systems monitor
and log how and when the user plays the game. As a consequence, the game
provider can fine-tune the difficulty of the game, suggest other related games
that would most likely interest the user, and offer additional equipment and
enhancements for the character based on the user’s age, gender, and interests.
This information may get stored locally or uploaded to the game provider’s
cloud to analyze the gaming habits and opportunities for upsell and cross-sell,
and identify archetypical profiles of specific kinds of users.

Smartphones provide another rich source of data. In addition to messaging and


basic phone usage, they store and transmit data about Internet usage, SMS
usage, and real-time location. This metadata can be used for analyzing traffic
patterns by scanning the density of smartphones in locations to track the speed
of cars or the relative traffic congestion on busy roads. In this way, GPS devices
in cars can give drivers real-time updates and offer alternative routes to avoid
traffic delays.

Retail shopping loyalty cards record not just the amount an individual spends,
but the locations of stores that person visits, the kinds of products purchased,
the stores where goods are purchased most often, and the combinations of
products purchased together. Collecting this data provides insights into
shopping and travel habits and the likelihood of successful advertisement
targeting for certain types of retail promotions.
Data collectors include sample entities that collect data from the device and
users.

Data results from a cable TV provider tracking the shows a person watches,
which TV channels someone will and will not pay for to watch on demand, and
the prices someone is willing to pay for premium TV content

Retail stores tracking the path a customer takes through their store while
pushing a shopping cart with an RFID chip so they can gauge which products
get the most foot traffic using geospatial data collected from the RFID chips

Data aggregators make sense of the data collected from the various entities
from the “SensorNet” or the “Internet of

Things.” These organizations compile data from the devices and usage patterns
collected by government agencies, retail stores, and websites. In turn, they can
choose to transform and package the data as products to sell to list brokers, who
may want to generate marketing lists of people who may be good targets for
specific ad campaigns.

Data users and buyers

These groups directly benefit from the data collected and aggregated by others
within the data value chain.

Retail banks, acting as a data buyer, may want to know which customers have
the highest likelihood to apply for a second mortgage or a home equity line of
credit. To provide input for this analysis, retail banks may purchase data from a
data aggregator. This kind of data may include demographic information about
people living in specific locations; people who appear to have a specific level of
debt, yet still have solid credit scores (or other characteristics such as paying
bills on time and having savings accounts) that can be used to infer credit
worthiness; and those who are searching the web for information about paying
off debts or doing home remodeling projects.

Obtaining data from these various sources and aggregators will enable a more
targeted marketing campaign, which would have been more challenging before
Big Data due to the lack of information or high-performing technologies.

Using technologies such as Hadoop to perform natural language processing on


unstructured, textual data from social media websites, users can gauge the
reaction to events such as presidential campaigns. People may, for example,
want to determine public sentiments toward a candidate by analyzing related
blogs and online comments. Similarly, data users may want to track and prepare
for natural disasters by identifying which areas a hurricane affects first and how
it moves, based on which geographic areas are tweeting about it or discussing it
via social media.

A field to analyze and to extract information about the big data involved in the
business or the data world so that proper conclusions can be made is called big
data Analytics.

These conclusions can be used to predict the future or to forecast the business.

Also, this helps in creating a trend about the past.

Skilled professionals in statistics and engineering with domain knowledge are


needed in the analysis of big data as the data is huge, and analysis needs proper
determination and skillset.

This data is more complex that it cannot be dealt with with traditional methods
of analysis.

We produce a massive amount of data each day, whether we know about it or


not. Every click on the internet, every bank transaction, every video we watch
on YouTube, every email we send, every like on our Instagram post makes up
data for tech companies.

With such a massive amount of data being collected, it only makes sense for
companies to use this data to understand their customers and
their behavior better. This is the reason why the popularity of Data Science
has grown manifold over the last few years.

A big data platform is a type of IT solution that combines the features and
capabilities of several big data applications and utilities within a single
solution, this is then used further for managing as well as analyzing Big Data.
It focuses on providing its users with efficient analytics tools for massive
datasets.
The users of such platforms can custom build applications according to their
use case like to calculate customer loyalty (E-Commerce user case), and so
on.
Goal: The main goal of a Big Data Platform is to achieve: Scalability,
Availability, Performance, and Security.
Example: Some of the most commonly used Big Data Platforms are :

 Hadoop Delta Lake Migration Platform


 Data Catalog Platform
 Data Ingestion Platform
 IoT Analytics Platform
 Drivers for Big Data
 Big Data has quickly risen to become one of the most desired topics in
the industry.
 The main business drivers for such rising demand for Big Data Analytics
are :
1. The digitization of society
2. The drop in technology costs
3. Connectivity through cloud computing
4. Increased knowledge about data science
5. Social media applications
6. The rise of Internet-of-Things(IoT)
Example: A number of companies that have Big Data at the core of their
strategy like :
Apple, Amazon, Facebook and Netflix have become very successful at
the beginning of the 21st century.
 Big Data Architecture :
 Big data architecture is designed to handle the ingestion, processing,
and analysis of data that is too large or complex for traditional database
systems.

 The big data architectures include the following components:


 Data sources: All big data solutions start with one or more data
sources.
 Example,
 Application data stores, such as relational databases.
 Static files produced by applications, such as web server log files.
 Real-time data sources, such as IoT devices.
 Data storage: Data for batch processing operations is stored in a
distributed file store that can hold high volumes of large files in various
formats (also called data lake).
 Example,
 Azure Data Lake Store or blob containers in Azure Storage.
 Batch processing: Since the data sets are so large, therefore a big
data solution must process data files using long-running batch jobs to
filter, aggregate, and prepare the data for analysis.
 Real-time message ingestion: If a solution includes real-time sources,
the architecture must include a way to capture and store real-time
messages for stream processing.
 Stream processing: After capturing real-time messages, the solution
must process them by filtering, aggregating, and preparing the data for
analysis. The processed stream data is then written to an output sink.
We can use open-source Apache streaming technologies like Storm
and Spark Streaming for this.
 Analytical data store: Many big data solutions prepare data for
analysis and then serve the processed data in a structured format that
can be queried using analytical tools. Example: Azure Synapse
Analytics provides a managed service for large-scale, cloud-based data
warehousing.
 Analysis and reporting: The goal of most big data solutions is to
provide insights into the data through analysis and reporting. To
empower users to analyze the data, the architecture may include a data
modelling layer. Analysis and reporting can also take the form of
interactive data exploration by data scientists or data analysts.
 Orchestration: Most big data solutions consist of repeated data
processing operations, that transform source data, move data between
multiple sources and sinks, load the processed data into an analytical
data store, or push the results straight to a report. To automate these
workflows, we can use an orchestration technology such as Azure Data
Factory.

Applications of Big Data


In today’s world big data have several applications, some of them are listed
below :
Tracking Customer Spending Habit, Shopping Behavior :
In big retails stores, the management team has to keep data of customer’s
spending habits, shopping behaviour, most liked product, which product is
being searched/sold most, based on that data, the production/collection rate of
that product gets fixed.
Recommendation :
By tracking customer spending habits, shopping behaviour, big retail stores
provide recommendations to the customers.
Smart Traffic System :
Data about the condition of the traffic of different roads, collected through
cameras, GPS devices placed in the vehicle.
All such data are analyzed and jam-free or less jam way, less time taking
ways are recommended.
One more profit is fuel consumption can be reduced.
Secure Air Traffic System :
At various places of flight, sensors are present.
These sensors capture data like the speed of flight, moisture, temperature,
and other environmental conditions.
Based on such data analysis, an environmental parameter within flight is set
up and varied.
By analyzing flight’s machine-generated data, it can be estimated how long
the machine can operate flawlessly and when it can be replaced/repaired.
Auto Driving Car :
In the various spots of the car camera, a sensor is placed that gathers data
like the size of the surrounding car, obstacle, distance from those, etc.
These data are being analyzed, then various calculations are carried out.
These calculations help to take action automatically.
Virtual Personal Assistant Tool :
Big data analysis helps virtual personal assistant tools like Siri, Cortana and
Google Assistant to provide the answer to the various questions asked by
users.
This tool tracks the location of the user, their local time, season, other data
related to questions asked, etc.
Analyzing all such data provides an answer.
Example: Suppose one user asks “Do I need to take Umbrella?”The tool
collects data like location of the user, season and weather condition at that
location, then analyzes these data to conclude if there is a chance of raining,
then provides the answer.
IoT :
Manufacturing companies install IOT sensors into machines to collect
operational data.
Analyzing such data, it can be predicted how long a machine will work without
any problem when it requires repair.
Thus, the cost to replace the whole machine can be saved.
Education Sector Energy Sector :
Online educational courses conducting organization utilize big data to search
candidates interested in that course.
If someone searches for a YouTube tutorial video on a subject, then an online
or offline course provider organization on that subject sends an ad online to
that person about their course.
Media and Entertainment Sector :
Media and entertainment service providing company like Netflix,
Amazon Prime, Spotify do analysis on data collected from their users.
Data like what type of video, music users are watching, listening to most,
how long users are spending on site, etc are collected and analyzed to set
the next business strategy.

Traditional BI versus Big data

Traditional Data Big Data

Traditional data is generated in Big data is generated outside the


enterprise level. enterprise level.

Its volume ranges from Gigabytes to Its volume ranges from Petabytes to
Terabytes. Zettabytes or Exabytes.
Traditional Data Big Data

Big data system deals with


structured, semi-
Traditional database system deals with structured,database, and
structured data. unstructured data.

Traditional data is generated per hour But big data is generated more
or per day or more. frequently mainly per seconds.

Traditional data source is centralized Big data source is distributed and it


and it is managed in centralized form. is managed in distributed form.

Data integration is very easy. Data integration is very difficult.

Normal system configuration is capable High system configuration is


to process traditional data. required to process big data.

The size is more than the traditional


The size of the data is very small. data size.

Special kind of data base tools are


Traditional data base tools are required required to perform any
to perform any data base operation. databaseschema-based operation.

Special kind of functions can


Normal functions can manipulate data. manipulate data.

Its data model is strict schema based Its data model is a flat schema
and it is static. based and it is dynamic.

Traditional data is stable and inter Big data is not stable and unknown
relationship. relationship.

Traditional data is in manageable Big data is in huge volume which


volume. becomes unmanageable.

It is easy to manage and manipulate It is difficult to manage and


the data. manipulate the data.

Its data sources includes ERP Its data sources includes social
Traditional Data Big Data

transaction data, CRM transaction data,


financial data, organizational data, web media, device data, sensor data,
transaction data etc. video, images, audio etc.

A typical data warehouse often includes the following elements:


 A relational database to store and manage data
 An extraction, loading, and transformation (ELT) solution for preparing the data for
analysis
 Statistical analysis, reporting, and data mining capabilities
 Client analysis tools for visualizing and presenting data to business users

A typical Hadoop environment


Hadoop is an Apache open source framework written in java that allows distributed processing of
large datasets across clusters of computers using simple programming models. The Hadoop
framework application works in an environment that provides
distributed storage and computation across clusters of computers. Hadoop is designed to scale up
from single server to thousands of machines, each offering local computation and storage.

Hadoop Architecture
At its core, Hadoop has two major layers namely −

 Processing/Computation layer (MapReduce), and


 Storage layer (Hadoop Distributed File System).
MapReduce
MapReduce is a parallel programming model for writing distributed applications devised at
Google for efficient processing of large amounts of data (multi-terabyte data-sets), on large
clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner. The
MapReduce program runs on Hadoop which is an Apache open-source framework.

Hadoop Distributed File System


The Hadoop Distributed File System (HDFS) is based on the Google File System (GFS) and
provides a distributed file system that is designed to run on commodity hardware. It has many
similarities with existing distributed file systems. However, the differences from other
distributed file systems are significant. It is highly fault-tolerant and is designed to be
deployed on low-cost hardware. It provides high throughput access to application data and is
suitable for applications having large datasets.
Apart from the above-mentioned two core components, Hadoop framework also includes the
following two modules −
 Hadoop Common − These are Java libraries and utilities required by other
Hadoop modules.
 Hadoop YARN − This is a framework for job scheduling and cluster resource
management.
How Does Hadoop Work?
It is quite expensive to build bigger servers with heavy configurations that handle large scale
processing, but as an alternative, you can tie together many commodity computers with
single-CPU, as a single functional distributed system and practically, the clustered machines
can read the dataset in parallel and provide a much higher throughput. Moreover, it is cheaper
than one high-end server. So this is the first motivational factor behind using Hadoop that it
runs across clustered and low-cost machines.
Hadoop runs code across a cluster of computers. This process includes the following core
tasks that Hadoop performs −
 Data is initially divided into directories and files. Files are divided into
uniform sized blocks of 128M and 64M (preferably 128M).
 These files are then distributed across various cluster nodes for further
processing.
 HDFS, being on top of the local file system, supervises the processing.
 Blocks are replicated for handling hardware failure.
 Checking that the code was executed successfully.
 Performing the sort that takes place between the map and reduce stages.
 Sending the sorted data to a certain computer.
 Writing the debugging logs for each job.

Advantages of Hadoop
 Hadoop framework allows the user to quickly write and test distributed
systems. It is efficient, and it automatic distributes the data and work across the
machines and in turn, utilizes the underlying parallelism of the CPU cores.
 Hadoop does not rely on hardware to provide fault-tolerance and high
availability (FTHA), rather Hadoop library itself has been designed to detect
and handle failures at the application layer.
 Servers can be added or removed from the cluster dynamically and Hadoop
continues to operate without interruption.
 Another big advantage of Hadoop is that apart from being open source, it is
compatible on all the platforms since it is Java based.

A coexistence of data warehouse and Hadoop environment

You might also like