lOMoARcPSD|43134458

UNIT V Hadoop Related Tools

Artificial Intelligence (Dhanalakshmi College of Engineering)

Scan to open on Studocu

Studocu is not sponsored or endorsed by any college or university

Downloaded by Sristi Barman ([email protected])
 lOMoARcPSD|43134458

UNIT V HADOOP RELATED TOOLS

Hbase – data model and implementations – Hbase clients – Hbase examples – praxis.
Pig – Grunt – pig data model – Pig Latin – developing and testing Pig Latin scripts. Hive
– data types and file formats – HiveQL data definition – HiveQL data manipulation –
HiveQL queries.

Hbase
● Hbase is an open source and sorted map data built on Hadoop. It is column
oriented and horizontally scalable.
● It is based on Google's Big Table. It has a set of tables which keep data in key
value format.
● Hbase is well suited for sparse data sets which are very common in big data use
cases.
● Hbase provides APIs enabling development in practically any programming
language.
● It is a part of the Hadoop ecosystem that provides random real-time read/write
access to data in the Hadoop File System.

Features of Hbase

● Horizontally scalable: You can add any number of columns at anytime.

● Automatic Failover: Automatic failover is a resource that allows a system
administrator to automatically switch data handling to a standby system in the
event of system compromise.
● Integrations with Map/Reduce framework: All the commands and java codes
internally implemented in Map/ Reduce to do the task and it is built over the
Hadoop Distributed File System.

HBase Architecture:

HBase architecture has 3 main components: HMaster, Region Server, Zookeeper.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Figure – Architecture of HBase

All the 3 components are described below:

● HMaster –
1)The implementation of Master Server in HBase is HMaster. It is a process in
which regions are assigned to the region server as well as DDL (create, delete
table) operations. It monitors all Region Server instances present in the cluster.
In a distributed environment, Master runs several background threads. HMaster
has many features like controlling load balancing, failover etc.
● Region Server –
HBase Tables are divided horizontally by row key range into Regions. Regions
are the basic building elements of HBase cluster that consists of the distribution
of tables and are composed of Column families. Region Server runs on HDFS
DataNode which is present in the Hadoop cluster. Regions of Region Server are
responsible for several things, like handling, managing, executing as well as
reads and writes of HBase operations on that set of regions. The default size of a
region is 256 MB.
● Zookeeper –
It is like a coordinator in HBase. It provides services like maintaining
configuration information, naming, providing distributed synchronization, server
failure notification etc. Clients communicate with region servers via zookeeper.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

HDFS:
● HDFS is a Hadoop distributed File System, as the name implies it provides a
distributed environment for the storage and it is a file system designed in a way
to run on commodity hardware.
● It stores each file in multiple blocks and to maintain fault tolerance, the blocks are
replicated across a Hadoop cluster.
● HDFS provides a high degree of fault –tolerance and runs on cheap commodity
hardware. By adding nodes to the cluster and performing processing & storing by
using the cheap commodity hardware, it will give the client better results as
compared to the existing one.
● Here, the data stored in each block replicates into 3 nodes. In a case when any
node goes down there will be no loss of data, it will have a proper backup
recovery mechanism.
● HDFS gets in contact with the HBase components and stores a large amount of
data in a distributed manner.

HBase Data Model

● HBase Data Model is a set of components that consists of Tables, Rows, Column
families, Cells, Columns, and Versions.
● HBase tables contain column families and rows with elements defined as Primary
keys.
● A column in the HBase data model table represents attributes to the objects.
● HBase Data Model consists of following elements,
1) Set of tables: Each table with column families and rows
2) Each table must have an element defined as Primary Key.
3) Row key acts as a Primary key in HBase.
4) Any access to HBase tables uses this Primary Key
5) Each column present in HBase denotes attribute corresponding to object

Storage Mechanism in HBase

● HBase is a column-oriented database and data is stored in tables. The tables are
sorted by RowId.
● As shown below, HBase has RowId, which is the collection of several column
families that are present in the table.
● The column families that are present in the schema are key-value pairs. If we
observe in detail each column family having multiple numbers of columns.
● The column values stored into disk memory. Each cell of the table has its own
Metadata like timestamp and other information.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

● Coming to HBase the following are the key terms representing table schema
1) Table: Collection of rows present.
2) Row: Collection of column families.
3) Column Family: Collection of columns.
4) Column: Collection of key-value pairs.
5) Namespace: Logical grouping of tables.
6) Cell: A {row, column, version} tuple exactly specifies a cell definition in
HBase.

Column-oriented vs Row-oriented storages

Column and Row-oriented storages differ in their storage mechanism. As we all know
traditional relational models store data in terms of row-based format like in terms of
rows of data. Column-oriented storages store data tables in terms of columns and
column families.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Row-Oriented Database Column-Oriented Database

Data is stored and retrieved one row at In this type of data store, data is
a time and hence could read stored and retrieved in columns and
unnecessary data if some of the data in hence it is only able to read only the
a row is required. relevant data if required.

In this type of data store, read and

Records in Row Oriented Data stores
write operations are slower as
are easy to read and write.
compared to row-oriented.

Column-oriented stores are best

Row-oriented data stores are best
suited for online analytical
suited for online transaction systems.
processing.

These are not efficient in performing These are efficient in performing

operations applicable to the entire operations applicable to the entire
datasets and hence aggregation in dataset and hence enable
row-oriented is an expensive job or aggregation over many rows and
operation. columns.

HBase Implementations
HBase Read and Write Data Explained
The Read and Write operations from Client into Hfile can be shown in the diagram
below.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Step 1) Client wants to write data and in turn first communicates with Regions server
and then regions.
Step 2) Regions contacting memstore for storing associated with the column family
Step 3) First data stores into Memstore, where the data is sorted and after that, it
flushes into HFile. The main reason for using Memstore is to store data in a Distributed
file system based on Row Key. Memstore will be placed in Region server main memory
while HFiles are written into HDFS.
Step 4) Client wants to read data from Regions
Step 5) In turn Client can have direct access to Mem store, and it can request for data.
Step 6) Client approaches HFiles to get the data. The data is fetched and retrieved by
the Client. Memstore holds in-memory modifications to the store.

The hierarchy of objects in HBase Regions is as shown from top to bottom in the below
table.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Table HBase table present in the HBase cluster

Region HRegions for the presented tables

Store It stores per ColumnFamily for each

region for the table

Memstore ● Memstore for each store for each

region for the table
● It sorts data before flushing into
HFiles
● Write and read performance will
increase because of sorting

StoreFile StoreFiles for each store for each region

for the table

Block Blocks present inside StoreFiles

HBase vs. HDFS

HBase runs on top of HDFS and Hadoop. Some key differences between HDFS and
HBase are in terms of data operations and processing.

HBASE HDFS

Low latency operations High latency operations

Random reads and writes Write once Read many times

Accessed through shell commands, client Primarily accessed through MR (Map

API in Java, REST, Avro or Thrift Reduce) jobs

Storage and process both can be perform It’s only for storage areas

Hbase clients:
There are a number of client options for interacting with an HBase cluster.
Java HBase, like Hadoop, is written in Java. Example 20-1 shows the Java version of
how you would do the shell operations listed in the previous section. Example 20-1.
Basic table administration and access

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

● The class has a main() method and uses the HBaseConfiguration class to create
a Configuration object that reads the HBase configuration from the
hbase-site.xml and hbase-default.xml files.
● The Configuration object is used to create instances of HBaseAdmin and HTable,
which are used for administering the HBase cluster and accessing a specific
table, respectively.
● The code creates a table named "test" with a single column family named "data"
and asserts that the table was created. It then inserts data into the table using

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

● Put objects and retrieves and prints the first row using a Get object and scans
over the table using a Scan object.
● Finally, the code disables and deletes the table. The code makes use of HBase's
Bytes utility class to convert identifiers and values to the byte arrays that HBase
requires.

Praxis:
● Praxis is a project in HBase that aims to provide a common data ingestion
framework.
● Its main role is to import data from many different data sources (such as
relational databases, file systems, web pages, etc.) into HBase for analysis and
processing.
● With Praxis, users can define the mapping between the data source and the
target HBase table with simple configuration, and specify data transformation and
filtering rules.
● Praxis also provides a set of tools and APIs for data import, making it easy to
import unstructured or structured data into HBase. Best practices for using Praxis
include:
1. Before using Praxis, make sure you have a clear understanding of the
structure and contents of the data source and how to map it to HBase
tables.
2. When writing configuration files, try to keep the configuration concise and
easy to understand. You can manipulate data in a data source by using
regular expressions, filters, and so on.
3. In the process of data import, you can use the components and APIs
provided by Praxis to realize incremental import or full import of data to
meet different needs.
4. Pay attention to the format and encoding of the data source to ensure that
the data can be correctly converted and imported into HBase.
5. When importing large-scale data, consider using Praxis' parallel import
function to speed up the import and reasonably configure the number of
concurrent tasks and thread pool size for import.

Pig

● Pig Hadoop is basically a high-level programming language that is helpful for the
analysis of huge datasets.
● Pig Hadoop was developed by Yahoo! and is generally used with Hadoop to
perform a lot of data administration operations.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

● For writing data analysis programs, Pig renders a high-level programming

language called Pig Latin.
● Several operators are provided by Pig Latin using which personalized functions
for writing, reading, and processing of data can be developed by programmers.
● For analyzing data through Apache Pig, we need to write scripts using Pig Latin.
Then, these scripts need to be transformed into MapReduce tasks. This is
achieved with the help of Pig Engine.

Apache Pig Architecture

The main reason why programmers have started using Hadoop Pig is that it converts
the scripts into a series of MapReduce tasks making their job easy. Below is the
architecture of Pig Hadoop:

Pig Hadoop framework has four main components:

1. Parser: When a Pig Latin script is sent to Hadoop Pig, it is first handled by
the parser. The parser is responsible for checking the syntax of the script,
along with other miscellaneous checks. Parser gives an output in the form of
a Directed Acyclic Graph (DAG) that contains Pig Latin statements, together
with other logical operators represented as nodes.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

2. Optimizer: After the output from the parser is retrieved, a logical plan for
DAG is passed to a logical optimizer. The optimizer is responsible for
carrying out the logical optimizations.
3. Compiler: The role of the compiler comes in when the output from the
optimizer is received. The compiler compiles the logical plan sent by the
optimizer. The logical plan is then converted into a series of MapReduce
tasks or jobs.
4. Execution Engine: After the logical plan is converted to MapReduce jobs,
these jobs are sent to Hadoop in a properly sorted order, and these jobs are
executed on Hadoop for yielding the desired result.

Features of Pig Hadoop

There are several features of Apache Pig:

1. In-built operators: Apache Pig provides a very good set of operators for
performing several data operations like sort, join, filter, etc.
2. Ease of programming: Since Pig Latin has similarities with SQL, it is very
easy to write a Pig script.
3. Automatic optimization: The tasks in Apache Pig are automatically
optimized. This makes the programmers concentrate only on the semantics
of the language.
4. Handles all kinds of data: Apache Pig can analyze both structured and
unstructured data and store the results in HDFS.

Grunt
● Grunt is a JavaScript-based task runner that provides line-editing facilities similar
to those found in GNU Readline, used in the bash shell and many other
command-line applications.
● It offers features such as command history, line recall, and a completion
mechanism.
● For instance, the Ctrl-E key combination moves the cursor to the end of the line,
and Ctrl-P or Ctrl-N (or the up or down cursor keys) can be used to recall lines in
the history buffer.
● The Tab key triggers Grunt's completion mechanism, which attempts to complete
Pig Latin keywords and functions.
● Customizing the completion tokens is also possible by creating a file named
autocomplete.
● When the Grunt session is finished, it can be exited with the quit command or the
equivalent shortcut \q.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Pig data model:

● The Pig data model in Hadoop is based on a high-level data processing
language called Pig Latin, which provides a rich set of data types and operators
to perform various operations on data
● The architecture of Apache Pig consists of several components, including a
parser, logical optimizer, compiler, and execution engine. Pig's data model can be
divided into two categories: scalar types and complex types
● Scalar types include:
1) Int: An integer
2) long: A long integer.
3) float: A floating-point number.
● Complex types include:
1) Maps: A collection of key-value pairs.
2) tuples: A fixed-length sequence of fields.
3) bags: A variable-length sequence of fields.
● Pig's data model allows for the analysis of both structured and unstructured data,
and it can handle missing data values
● The language is designed to be extensible, self-optimizing, and easy to use for
developers, even those who are not familiar with Java programming.
● To process data stored in HDFS, programmers write scripts using Pig Latin,
which are then converted into a series of MapReduce jobs by the Pig Engine.
● The result of the Pig execution is stored in HDFS, and the data structure can be
multivalued, nested, and richer

Pig Latin:
● Pig Latin is a high-level scripting language used in Apache Pig, which is a
platform for analyzing large datasets that runs on Apache Hadoop.
● It provides a high-level of abstraction for processing over MapReduce and allows
users to express their data analysis programs in a textual language called Pig
Latin.
● This language is designed to make MapReduce programming high level, similar
to that of SQL for relational database management systems.
● It can be extended using user-defined functions (UDFs) written in Java, Python,
JavaScript, Ruby, or Groovy and can execute Hadoop jobs in MapReduce,
Apache Tez, or Apache Spark.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Hive
● Apache Hive is a data warehouse software project that is built on top of the
Hadoop ecosystem.
● It provides an SQL-like interface to query and analyze large datasets stored in
Hadoop’s distributed file system (HDFS) or other compatible storage systems.
● Hive uses a language called HiveQL, which is similar to SQL, to allow users to
express data queries, transformations, and analyses in a familiar syntax.
● HiveQL statements are compiled into MapReduce jobs, which are then executed
on the Hadoop cluster to process the data.
● Hive includes many features that make it a useful tool for big data analysis,
including support for partitioning, indexing, and user-defined functions (UDFs).
● It also provides a number of optimization techniques to improve query
performance, such as predicate pushdown, column pruning, and query
parallelization.
● Hive can be used for a variety of data processing tasks, such as data
warehousing, ETL (extract, transform, load) pipelines, and ad-hoc data analysis.
Architecture of Hive:

Hive chiefly consists of three core parts:

● Hive Clients: Hive offers a variety of drivers designed for communication with
different applications. For example, Hive provides Thrift clients for Thrift-based
applications. These clients and drivers then communicate with the Hive server,
which falls under Hive services.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

● Hive Services: Hive services perform client interactions with Hive. For
example, if a client wants to perform a query, it must talk with Hive services.
● Hive Storage and Computing: Hive services such as file system, job client,
and meta store then communicates with Hive storage and stores things like
metadata table information and query results.
● The Metastore: The metastore is the central repository of Hive metadata. The
metastore is divided into two pieces: a service and the backing store for the
data. By default, the metastore service runs in the same JVM as the Hive
service and contains an embedded Derby database instance backed by the
local disk. This is called the embedded metastore configuration. Using an
embedded metastore is a simple way to get started with Hive; however, only
one embedded Derby database can access the database files on disk at any
one time, which means you can have only one Hive session open at a time
that accesses the same metastore. Trying to start a second session produces
an error when it attempts to open a connection to the metastore.

Features of Hive

○ Hive is fast and scalable.

○ It provides SQL-like queries (i.e., HQL) that are implicitly transformed to
MapReduce or Spark jobs.
○ It is capable of analyzing large datasets stored in HDFS.
○ It allows different storage types such as plain text, RCFile, and HBase.
○ It uses indexing to accelerate queries.
○ It can operate on compressed data stored in the Hadoop ecosystem.
○ It supports user-defined functions (UDFs) where users can provide its
functionality.

Limitations of Hive

○ Hive is not capable of handling real-time data.

○ It is not designed for online transaction processing.
○ Hive queries contain high latency.

Differences between Hive and Pig

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Hive Pig

Hive is commonly used by Data Analysts. Pig is commonly used by programmers.

It follows SQL-like queries. It follows the data-flow language.

It can handle structured data. It can handle semi-structured data.

It works on the server-side of the HDFS It works on the client-side of the HDFS
cluster. cluster.

Hive is slower than Pig. Pig is comparatively faster than Hive.

Hive data types:

Hive supports both primitive and complex data types. Primitives include numeric,
Boolean, string, and timestamp types. The complex data types include arrays, maps,
and structs.

Primitive types:
● BOOLEAN type for storing true and false values.
● There are four signed integral types: TINYINT, SMALLINT, INT, and BIGINT,
which are equivalent to Java’s byte, short, int, and long primitive types,
respectively (they are 1-byte, 2-byte, 4-byte, and 8-byte signed integers).
● Hive’s floating-point types, FLOAT and DOUBLE, correspond to Java’s float and
double, which are 32-bit and 64-bit floating-point numbers.
● The DECIMAL data type is used to represent arbitrary-precision decimals.
DECIMAL values are stored as unscaled integers.
● There are three Hive data types for storing text. STRING is a variable-length
character string with no declared maximum length. (The theoretical maximum
size STRING that may be stored is 2 GB.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

● The BINARY data type is for storing variable-length binary data.

● The TIMESTAMP data type stores timestamps with nanosecond precision. Hive
comes with UDFs for converting between Hive timestamps, Unix timestamps
(seconds since the Unix epoch), and strings, which makes most common date
operations tractable.
● The DATE data type stores a date with year, month, and day components.

Complex types:
● Hive has four complex types: ARRAY, MAP, STRUCT, and UNION. ARRAY and
MAP.
● STRUCT is a record type that encapsulates a set of named fields.
● A UNION specifies a choice of data types; values must match exactly one of
these types.
● Complex types permit an arbitrary level of nesting. Complex type declarations
must specify the type of the fields in the collection, using an angled bracket
notation.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Hive File Formats:

Apache Hive supports several familiar file formats used in Apache Hadoop. Hive can
load and query different data files created by other Hadoop components such as Pig or
MapReduce.
Different file formats and compression codecs work better for different data sets in
Apache Hive.
Following are the Apache Hive different file formats:

● Text File
● Sequence File
● RC File
● AVRO File
● ORC File
● Parquet File

Hive Text File Format

● Hive Text file format is a default storage format. You can use the text format to
interchange the data with other client applications. The text file format is very
common in most of the applications. Data is stored in lines, with each line being a
record. Each line is terminated by a newline character (\n).
● The text format is a simple plane file format. You can use the compression
(BZIP2) on the text file to reduce the storage spaces.
● Create a TEXT file by adding the storage option as ‘STORED AS TEXTFILE’ at
the end of a Hive CREATE TABLE command.
● Example:

Create table textfile_table

(column_specs)

stored as textfile;

Hive Sequence File Format:

● Sequence files are Hadoop flat files which store values in binary key-value pairs.
The sequence files are in binary format and these files are able to split. The main
advantage of using sequence files is to merge two or more files into one file.
● Create a sequence file by adding the storage option as ‘STORED AS
SEQUENCEFILE’ at the end of a Hive CREATE TABLE command.
● Example:
Create table sequencefile_table

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

(column_specs)

stored as sequencefile;

Hive RC File Format

● RCFile is row columnar file format. This is another form of Hive file format which
offers high row level compression rates. If you have a requirement to perform
multiple rows at a time then you can use RCFile format.
● The RCFile is very much similar to the sequence file format. This file format also
stores the data as key-value pairs.
● Create RCFile by specifying ‘STORED AS RCFILE’ option at the end of a
CREATE TABLE Command
● Example:
Create table RCfile_table
(column_specs)

stored as rcfile;

Hive AVRO File Format

● AVRO is an open source project that provides data serialization and data
exchange services for Hadoop. You can exchange data between the Hadoop
ecosystem and programs written in any programming language. Avro is one of
the popular file formats in Big Data Hadoop based applications.
● Create AVRO file by specifying ‘STORED AS AVRO’ option at the end of a
CREATE TABLE Command.
● Example:
Create table avro_table
(column_specs)

stored as avro;

Hive ORC File Format

● The ORC file stands for Optimized Row Columnar file format. The ORC file
format provides a highly efficient way to store data in the Hive table. This file
system was actually designed to overcome limitations of the other Hive file
formats. The Use of ORC files improves performance when Hive is reading,
writing, and processing data from large tables.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

● Create an ORC file by specifying ‘STORED AS ORC’ option at the end of a

CREATE TABLE Command.
● Example:
Create table orc_table
(column_specs)

stored as orc;

Hive Parquet File Format

● Parquet is a column-oriented binary file format. The parquet is highly efficient for
the types of large-scale queries. Parquet is especially good for queries scanning
particular columns within a particular table. The Parquet table uses compression
Snappy, gzip; currently Snappy by default.
● Create a Parquet file by specifying ‘STORED AS PARQUET’ option at the end of
a CREATE TABLE Command.
● Example:
Create table parquet_table
(column_specs)

stored as parquet;

HiveQL: Data definition

● HiveQL is the Hive query language. Like all SQL dialects in widespread use, it
doesn’t fully conform to any particular revision of the ANSI SQL standard.
● It is perhaps closest to MySQL’s dialect, but with significant differences. Hive
offers no support for row-level inserts, updates, and deletes.
● Hive doesn’t support transactions. Hive adds extensions to provide better
performance in the context of Hadoop and to integrate with custom extensions
and even external programs.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Databases in Hive:

● The Hive concept of a database is essentially just a catalog or namespace of

tables. However, they are very useful for larger clusters with multiple teams and
users, as a way of avoiding table name collisions.
● It’s also common to use databases to organize production tables into logical
groups.

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Hive: Data manipulation

Introduction to Hive DML commands

● Hive DML (Data Manipulation Language) commands are used to insert, update,
retrieve, and delete data from the Hive table once the table and database
schema has been defined using Hive DDL commands.
● The various Hive DML commands are:
1. LOAD
2. SELECT
3. INSERT
4. DELETE

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

5. UPDATE
6. EXPORT
7. IMPORT

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Downloaded by Sristi Barman ([email protected])

 lOMoARcPSD|43134458

Downloaded by Sristi Barman ([email protected])