BD Chapter 6

Important Question Answers

Q1. Explain Hbase in detail (Architecture, components, functions)
Ans:
Definition: Apache HBase is a distributed, column-oriented database built on top of
the Hadoop Distributed File System (HDFS). It is designed to manage large volumes of
sparse data and provides real-time read/write access, making it suitable for big data
applications that require random, real-time access to data. Unlike traditional relational
databases, HBase does not use SQL and is not a relational database, but it supports
Java-based applications and can integrate with Apache Avro, REST, and Thrift for
additional data interaction.

Fig: HBase Architecture

Key Components of HBase Architecture
1. Master Node (HBase Master):
o The master node is responsible for managing the HBase cluster, overseeing tasks
like region assignment, load balancing, and ensuring region servers are running
smoothly.
o In the architecture, the master node directs interactions between clients and
HBase, acting as the central orchestrator.
2. Region Servers:
o Region servers manage and store the actual data in HBase. Each server is
responsible for handling regions (horizontal partitions of tables), storing and
managing data for one or more tables.
o These servers interact directly with HDFS, where data is stored in a fault-tolerant
manner. Each region server communicates with the master node to register
regions, perform read/write operations, and handle data requests from clients.
3. Regions:
o HBase tables are divided into regions, which are the smallest units of distribution.
Each region holds a subset of table data and is managed by region servers.
o Regions are split as data grows, allowing HBase to scale horizontally by adding
more region servers to handle increasing amounts of data.
4. ZooKeeper:
o HBase relies on Apache ZooKeeper for distributed coordination. ZooKeeper
helps manage metadata for HBase, keeping track of active servers, regions, and
ensuring high availability.
o It acts as a centralized service to manage configurations and synchronize access,
allowing the HBase cluster to operate reliably even with multiple nodes and
servers.
5. HDFS (Hadoop Distributed File System):
o HDFS serves as the underlying file storage system for HBase. All data in HBase is
stored on HDFS, enabling fault tolerance and data redundancy.
o HDFS ensures data durability and distributed storage, which allows HBase to
handle large volumes of data while maintaining reliability.
Functions of HBase:
1. Data Storage: Column-oriented storage with column families for efficient access.
2. Real-Time Access: Supports low-latency reads and writes for quick data retrieval.
3. Scalability: Expands horizontally by adding region servers as data grows.
4. Fault Tolerance: Ensures high availability using HDFS and ZooKeeper.
5. Big Data Integration: Works with Hadoop tools like Hive and Spark for flexibility.
Q2. Explain Sqoop in detail (Architecture, components, functions)
Ans:
Apache Sqoop is a tool used to transfer bulk data between Hadoop and external data
stores, primarily relational databases (RDBMS) like MySQL and MS SQL Server. Its main
function is to facilitate the movement of data into and out of Hadoop Distributed File
System (HDFS) from RDBMS to support data processing in a Hadoop ecosystem.
Sqoop addresses challenges such as maintaining data consistency, handling large
volumes, and enabling efficient resource utilization by automating the transfer of data
from multiple sources.

Fig: Sqoop Architecture

The architecture of Sqoop involves several stages:
1. Client Request: A user submits an import or export command to Sqoop to move
data between Hadoop and RDBMS.
2. Connector Framework: Sqoop has connectors for major RDBMSs (e.g., MySQL,
SQL Server) that help interact with different types of databases.
3. MapReduce Job: Sqoop uses MapReduce for parallelizing data import/export jobs.
Multiple mappers perform parallel tasks to import data from RDBMS into HDFS or
export it back from HDFS to RDBMS.
4. YARN Integration: Sqoop runs on the YARN framework, which provides fault
tolerance and resource management during parallel data transfer.
Sqoop Functions:
1. Data Import: Transfers data from an RDBMS to HDFS. Sqoop can import entire
tables or specific subsets of data. It can also perform incremental imports, pulling
only the new or updated data.
2. Data Export: Moves processed data from HDFS back into RDBMS. This supports
transferring large volumes of processed data for further analysis.
3. Parallel Execution: By using multiple mappers, Sqoop enables parallel processing
to increase efficiency during data transfer.
4. Kerberos Security Integration: Supports secure authentication, allowing safe data
transfer over potentially insecure networks.
5. Incremental Loads: Allows only the new or modified data to be imported without
reloading the entire dataset, saving time and resources
Q3. Explain Spark in detail (Architecture, components, functions)
Ans:
Definition: Apache Spark is an open-source, fast, and general-purpose cluster-
computing framework that extends the MapReduce model to efficiently handle a wide
range of data processing tasks, such as batch processing, interactive queries, real-time
streaming, and machine learning. Spark's key feature is its in-memory computing,
which allows it to perform operations significantly faster than traditional disk-based
processing frameworks.

Fig: Apache Spark Architecture

Components of Apache Spark:
1. Spark Core: The foundation of Spark, responsible for basic functions like task
scheduling, memory management, fault recovery, and distributed storage.
2. Spark SQL: Allows users to perform SQL-based queries on structured and semi-
structured data. It introduces DataFrames and a query engine optimized for
performance.
3. Spark Streaming: Extends Spark Core to support real-time data processing. It
divides streaming data into small batches and processes them in near real-time.
4. MLlib (Machine Learning Library): Spark’s scalable machine learning library,
providing tools for common machine learning tasks like classification, regression,
clustering, and collaborative filtering.
5. GraphX: A distributed graph-processing framework that allows for graph
computation and analytics, such as PageRank and shortest paths, and provides a
powerful API for graph manipulation.

Functions:
• Batch Processing: Spark processes large-scale data in batches, using its distributed
and parallel architecture to handle extensive datasets efficiently.
• Real-Time Processing: Spark Streaming processes real-time data streams, making
it ideal for applications requiring live data processing, like fraud detection and social
media analytics.
• Machine Learning and Graph Analysis: Through MLlib and GraphX, Spark enables
complex analytical tasks, including predictive analytics and graph-based
computations, to be performed at scale.
Q4. Difference between Hbase and RDBMS.

Ans:
SR Aspect HBase RDMS
1 Data Model Column-oriented storage Row-oriented storage
2 Data Access Row-key based, supports random Primary and foreign key-based access
access
3 Data Volume Ideal for large, sparse datasets Optimized for structured, smaller
datasets
4 Schema Schema-less, flexible column Fixed schema with tables and columns
Flexibility families
5 Scaling Horizontally scalable across Mostly vertically scalable
distributed servers
6 Query NoSQL; no SQL support SQL-based query support
Language
7 ACID Not fully ACID compliant Fully ACID compliant
Compliance
8 Read/Write Fast reads/writes for large data Moderate speed with structured data
Speed sets
9 Transaction Limited transaction support Strong transaction support
10 Use Cases Real-time analytics, large OLTP systems, structured data
datasets processing

Acronym: DS QAR TU