Assignment 1

1. What do you mean by Big Data? How is it different from traditional data?

Big Data refers to large and complex datasets that exceed the processing capabilities of traditional
database systems. It is characterized by the "3 Vs": Volume (large amounts of data), Velocity (fast
generation and processing of data), and Variety (diverse types of data such as structured, unstructured,
and semi-structured). Unlike traditional data, which is typically small, structured, and stored in relational
databases, Big Data can come from diverse sources like social media, sensors, and logs. It requires
specialized tools like Hadoop and Spark to process and store data, often in distributed environments.
Traditional data, in contrast, is simpler, smaller, and stored in centralized systems like SQL databases.

2. What are the challenges of handling Big Data?

Managing Big Data presents multiple challenges:

• Storage: Storing massive datasets requires scalable, distributed file systems like HDFS.

• Data Processing: Handling large volumes of data in parallel requires powerful frameworks such
as MapReduce and Spark.

• Data Integration: Combining structured and unstructured data from diverse sources can be
complex.

• Scalability: The infrastructure must scale horizontally to accommodate growing data.

• Security: Protecting large datasets, especially sensitive information, requires robust encryption
and access control.

• Data Quality: Ensuring data accuracy, completeness, and consistency across sources is difficult.

• Cost: Maintaining Big Data infrastructure, including storage and processing resources, can be
expensive for enterprises.

Shruti Dutt 21100BTCSE09987

3. Describe the architecture of Hadoop.

Hadoop is a framework for distributed storage and processing of large datasets. It includes:

• HDFS (Hadoop Distributed File System): Stores data across a cluster of machines, breaking files
into blocks and replicating them for fault tolerance.

• MapReduce: A programming model that processes data in parallel by dividing tasks into smaller
sub-tasks, each handled by a different node in the cluster.

• YARN (Yet Another Resource Negotiator): Manages and schedules resources across the cluster,
ensuring efficient task execution.

• Other Tools: Tools like Hive (SQL interface), Pig (data transformation), HBase (NoSQL database),
and Zookeeper (coordination service) integrate with Hadoop to provide data management and
processing capabilities.

4. Write notes on MapReduce and HDFS in terms of Hadoop.

• MapReduce: A programming model used for processing large datasets in parallel. It consists of
two phases: the Map phase (data is processed into key-value pairs) and the Reduce phase
(aggregates results based on keys). MapReduce distributes the workload across a cluster,
allowing efficient parallel computation.

• HDFS: A distributed file system designed to store large datasets in blocks, distributed across
many nodes. HDFS provides fault tolerance by replicating each data block (usually three copies)
across different DataNodes. The NameNode manages metadata (file locations), while
DataNodes store the actual data. This architecture enables high scalability and reliability for Big
Data storage.

5. Define challenges associated with managing Big Data in an enterprise setting.

Enterprises face several challenges in managing Big Data:

• Data Governance: Ensuring data is accurate, compliant with regulations (e.g., GDPR), and
properly handled across the organization.

• Integration: Combining diverse datasets from different sources (databases, logs, social media)
can be complex and time-consuming.

• Scalability: As data grows, infrastructure must scale efficiently, which can be costly and
technically challenging.

• Security: Protecting sensitive data from breaches and ensuring secure access control are critical.

• Data Quality: Handling issues related to incomplete, inconsistent, or incorrect data

Shruti Dutt 21100BTCSE09987

6. Explain Name-Node and Data-Nodes in HDFS.

In HDFS, there are two key components:

• NameNode: The master node responsible for managing the metadata of the HDFS system. It
stores information about file names, directory structure, and where data blocks are located
across the cluster. However, it does not store actual data, only the metadata.

• DataNodes: These are the worker nodes that store the actual data blocks. Each DataNode is
responsible for managing the storage of data and handling read/write requests. DataNodes
periodically send heartbeats to the NameNode to indicate their health. If a DataNode fails, the
NameNode ensures that the data is replicated to other DataNodes to maintain fault tolerance.

7. How does HDFS ensure data reliability and fault tolerance?

HDFS ensures reliability and fault tolerance through several mechanisms:

• Replication: Each data block is replicated (default is three copies) across multiple DataNodes to
ensure redundancy.

• Heartbeat: DataNodes send regular heartbeats to the NameNode. If a DataNode fails to send a
heartbeat, it is marked as inactive, and the data is replicated from other healthy nodes.

• Data Integrity: Checksums are used to verify the integrity of the data stored on DataNodes. If
corruption is detected, HDFS attempts to recover the data from other replicas.

• Automatic Recovery: In case of DataNode failure, HDFS automatically replicates the missing data
from available replicas to ensure that data redundancy is maintained.

Shruti Dutt 21100BTCSE09987

 Assignment 2

1. What is Hive? Explain its anatomy. And outline the process of integrating Hive with Hadoop.

Hive is a data warehouse system built on top of Hadoop that enables users to perform SQL-like queries
(HiveQL) on large datasets stored in HDFS. Hive abstracts the complexity of MapReduce jobs, making it
easier to interact with Big Data.

• Anatomy of Hive:

o Metastore: Stores metadata such as table schemas and file locations.

o Driver: Interfaces with HiveQL and submits queries.

o Compiler: Converts HiveQL into MapReduce jobs.

o Execution Engine: Executes the MapReduce jobs across the Hadoop cluster.

Integration with Hadoop:

• Hive stores data in HDFS and processes queries by compiling them into MapReduce jobs. It
utilizes the Hadoop cluster for storage and parallel processing, enabling scalability for large data
sets.

2. Give the working of Zookeeper. What is the purpose of it in the Hadoop ecosystem?

Zookeeper is a distributed coordination service for managing configuration information, synchronization,

and naming across distributed systems. It maintains a tree-like structure of znodes, which can store data
and manage configuration parameters. Zookeeper provides:

• Synchronization: Ensures consistency and coordination between distributed nodes.

• Leader Election: Helps select a leader node in a distributed system, ensuring tasks are managed
efficiently.

• Configuration Management: Stores and manages configuration data for distributed systems.

In the Hadoop ecosystem, Zookeeper is crucial for managing distributed services such as HBase
(managing region servers) and Kafka (distributed messaging). It helps ensure fault tolerance and
consistency across these systems.

Shruti Dutt 21100BTCSE09987

3. What is MapReduce programming? Where is it used and why?

MapReduce is a programming model used for processing large datasets in parallel. It divides tasks into
two main phases:

• Map: Splits data into key-value pairs for parallel processing.

• Reduce: Aggregates results based on keys.

Where it is used:

• Data Analysis: Used for tasks like log analysis, data mining, and web indexing.

• ETL Processes: For large-scale data transformation and loading into data warehouses.

Why it is used:

• Scalability: It allows for efficient processing of Big Data by distributing tasks across a cluster.

• Fault Tolerance: It provides mechanisms for handling node failures, ensuring continuous
processing.

• Parallelism: MapReduce processes data in parallel, speeding up computations for large datasets.

4. What is the difference between Pig and Hive?

Pig and Hive are both high-level data processing tools in Hadoop, but they differ in their approach and
use cases:

• Pig uses Pig Latin, a procedural language that provides more flexibility for complex data
processing and transformations. It is suited for developers who need to write custom data flow
scripts.

• Hive uses HiveQL, a declarative SQL-like language that is easier for data analysts and SQL users
to work with. Hive abstracts the complexity of MapReduce, making it ideal for SQL-based
querying and data warehousing tasks.

In short, Pig is more flexible but requires a deeper understanding of programming, while Hive is simpler
and more suited for those familiar with SQL.

Shruti Dutt 21100BTCSE09987

5. Define the Hadoop ecosystem and list its core components.

The Hadoop ecosystem is a collection of tools and frameworks designed for storing and processing large
datasets in a distributed environment. Key components include:

• HDFS (Hadoop Distributed File System) for storing data.

• MapReduce for processing data in parallel across a cluster.

• YARN (Yet Another Resource Negotiator) for managing cluster resources.

Other tools in the ecosystem:

• Hive: Data warehousing and SQL-like querying.

• Pig: High-level scripting language for data transformation.

• HBase: NoSQL database for real-time data access.

• Zookeeper: Coordination and synchronization service for distributed systems.

• Oozie: Workflow scheduler for managing Hadoop jobs.

These components work together to enable scalable storage, processing, and management of Big Data.

6. Describe the installation steps for Apache Hadoop on a Linux system.

To install Apache Hadoop on a Linux system:

1. Install Java: Hadoop requires Java, so install it first.

2. Download Hadoop: Download the binary from the official Apache website.

3. Configure environment variables: Set HADOOP_HOME, JAVA_HOME, and other paths in your
.bashrc file.

4. Configure Hadoop: Edit core configuration files like core-site.xml, hdfs-site.xml, and mapred-
site.xml to set up HDFS and MapReduce.

5. Format HDFS: Run hdfs namenode -format to initialize the file system.

6. Start Hadoop: Start Hadoop daemons like NameNode, DataNode, ResourceManager, and
NodeManager.

Shruti Dutt 21100BTCSE09987

7. What is the significance of the HADOOP_HOME environment variable in Hadoop installation?

The HADOOP_HOME environment variable specifies the directory where Hadoop is installed. It is crucial
for Hadoop's operation because it helps the system locate the Hadoop binaries and configuration files.
By setting this variable, you ensure that Hadoop commands can be executed from anywhere in the
system and that services can access the correct configuration settings for the cluster. Additionally, other
environment variables like JAVA_HOME rely on the correct setup of HADOOP_HOME.

8. How does Apache Pig differ from MapReduce in the Hadoop ecosystem?

Pig and MapReduce are both used for processing Big Data, but they differ in their approach:

• Pig is a high-level data processing tool that uses Pig Latin, a scripting language designed for data
transformation and analysis. It provides an abstraction over MapReduce, making it easier to
write complex data processing tasks with fewer lines of code.

• MapReduce is a low-level, programming model that requires writing detailed code for both the
Map and Reduce phases, offering more flexibility but requiring more effort.

Pig simplifies data processing tasks, while MapReduce provides greater control but is more complex to
implement.

9. How is data loaded into Hadoop using HDFS dfs commands?

Data can be loaded into Hadoop's HDFS (Hadoop Distributed File System) using the hdfs dfs command-
line utility. This utility allows users to interact with HDFS to perform various operations like copying files,
listing directories, and viewing files. To load data into HDFS, the following steps are typically followed:

1. Copying Data to HDFS: To upload a local file into HDFS, you can use the -copyFromLocal or -put
command:

2. hdfs dfs -copyFromLocal /local/path/to/file /hdfs/destination/path

or

hdfs dfs -put /local/path/to/file /hdfs/destination/path

This command will copy the file from the local filesystem (/local/path/to/file) into the HDFS directory
(/hdfs/destination/path).

3. Verifying the Data: After copying the file, you can verify that the data was successfully uploaded
by listing the contents of the HDFS directory:

4. hdfs dfs -ls /hdfs/destination/path

5. Reading Data from HDFS: To read the content of a file stored in HDFS, use:

6. hdfs dfs -cat /hdfs/destination/path/filename

Shruti Dutt 21100BTCSE09987

 7. Additional Operations:

o You can also use -copyToLocal to copy data from HDFS to the local file system.

o The -rm command removes files or directories from HDFS.

These commands make it easy to load, manage, and interact with data within HDFS from the command
line.

10. Describe the role of HBase in handling large datasets within the Hadoop ecosystem.

HBase is a distributed, column-oriented NoSQL database built on top of Hadoop that provides real-time
read and write access to large datasets. It is designed to handle structured and semi-structured data that
is too large to fit into traditional relational databases. HBase plays an important role in the Hadoop
ecosystem by complementing HDFS for handling large-scale, low-latency data access.

Key features and role of HBase:

• Real-time Data Access: Unlike HDFS, which is optimized for batch processing, HBase allows for
real-time querying and updates, making it suitable for applications that require low-latency data
access.

• Column-family Storage: HBase stores data in column families, which allows for efficient retrieval
of specific columns rather than entire rows. This makes it highly efficient for certain types of
queries, especially on large datasets.

• Scalability: HBase can scale horizontally by adding more nodes to the cluster. This allows it to
handle petabytes of data across many machines without significant performance degradation.

• Fault Tolerance: Like HDFS, HBase also provides fault tolerance by replicating data across
multiple nodes. If a node fails, HBase ensures data availability and consistency through
replication.

• Integration with Hadoop Ecosystem: HBase integrates seamlessly with other Hadoop tools like
MapReduce, Hive, and Pig. It is often used in combination with HDFS for storing vast amounts of
data and with MapReduce for real-time analytics.

In summary, HBase is critical for applications that require quick, random read/write access to large
volumes of data in real-time, providing a flexible, scalable storage solution in the Hadoop ecosystem.

Shruti Dutt 21100BTCSE09987

 Assignment 3

1. What is the CAP theorem and its definition?

The CAP Theorem is a principle in distributed systems, introduced by Eric Brewer, which states that a
distributed database system can only guarantee two out of the following three properties at any given
time:

• Consistency: Every read operation will return the most recent write.

• Availability: Every request (read or write) will receive a response.

• Partition Tolerance: The system will continue to function despite network failures or
partitioning.

In practice, distributed systems must balance between consistency and availability when partitions occur,
as it's not possible to achieve all three simultaneously.

2. Explain the difference between SQL and NoSQL.

• SQL (Structured Query Language): Refers to relational databases (RDBMS), which use tables
(rows and columns) to store structured data. They follow a predefined schema and are typically
used for applications with complex queries, ACID transactions, and where data consistency is
crucial (e.g., MySQL, PostgreSQL).

• NoSQL (Not Only SQL): Refers to non-relational databases designed for flexible schema and
scalability. They handle unstructured or semi-structured data and can be used for real-time,
large-scale applications where high performance and horizontal scaling are required (e.g.,
MongoDB, Cassandra, Redis).

Shruti Dutt 21100BTCSE09987

3. Give an introduction to NoSQL. Write variations of NoSQL.

NoSQL databases are designed to provide flexible, scalable storage for large volumes of unstructured or
semi-structured data. These databases support different data models such as key-value pairs, document
stores, column families, or graphs. NoSQL databases are commonly used for Big Data, real-time analytics,
and applications with unpredictable data patterns.

Variations of NoSQL:

1. Document Stores: Store data in JSON-like documents (e.g., MongoDB, CouchDB).

2. Key-Value Stores: Store data as key-value pairs (e.g., Redis, DynamoDB).

3. Column-Family Stores: Organize data in columns instead of rows (e.g., HBase, Cassandra).

4. Graph Databases: Store data as graphs, which can represent relationships between entities (e.g.,
Neo4j, OrientDB).

4. What Querying Big Data with Hive?

Apache Hive is a data warehousing tool built on top of Hadoop that provides a SQL-like query interface
(HiveQL) for querying large datasets stored in HDFS. It abstracts the complexity of MapReduce by
translating HiveQL queries into MapReduce jobs.

Key Features:

• SQL-like Syntax: Allows for querying data without writing low-level MapReduce code.

• Schema-on-Read: Data is stored in raw form, and the schema is applied at the time of reading.

• Data Transformation: Hive supports data transformation, filtering, and aggregations using
HiveQL.

• Scalability: Hive is designed to handle petabytes of data using the distributed nature of Hadoop.

Shruti Dutt 21100BTCSE09987

5. Explain the steps involved in Moving the Data from RDBMS to Hadoop.

To move data from an RDBMS to Hadoop:

1. Data Extraction: Use Sqoop to extract data from relational databases like MySQL or PostgreSQL.

2. Data Transformation: Convert data into a format suitable for Hadoop (e.g., Parquet, Avro).

3. Data Loading: Load the data into HDFS using Sqoop or other ETL tools like Flume.

4. Data Processing: After loading into HDFS, use Hive or MapReduce to process the data.

5. Analysis: Finally, run analytics or queries using HiveQL or other Hadoop tools to derive insights.

6. How to move the Data from RDBMS to HBase?

To move data from RDBMS to HBase:

1. Install and Configure Sqoop: Install Sqoop on your Hadoop cluster and configure it to connect
with the RDBMS.

2. Create HBase Table: Define an HBase table with appropriate column families that match the
RDBMS data.

3. Map Data: Use Sqoop’s import command to map relational tables to HBase column families.

4. Data Import: Execute the Sqoop import command, which will read data from the RDBMS and
load it into HBase.

Example:

sqoop import --connect jdbc:mysql://localhost/db --table table_name --hbase-create-table --hbase-table

hbase_table --column-family cf_name

7. Explain the steps required to Move the Data from RDBMS to Hive.

Steps to move data from RDBMS to Hive:

1. Install and Configure Sqoop: Install Sqoop on your cluster and configure it with connection
details for the RDBMS.

2. Create Hive Table: Define a table in Hive that matches the schema of the RDBMS data.

3. Data Import: Use Sqoop’s --hive-import option to import data directly into Hive.

4. Verify Data: After import, use HiveQL to check and analyze the data within Hive.

Example:

sqoop import --connect jdbc:mysql://localhost/db --table table_name --hive-import --hive-table

hive_table_name

Shruti Dutt 21100BTCSE09987

 Assignment 4

1. What is the most useful library for Machine Learning in Python?

The most commonly used library for machine learning in Python is scikit-learn. It provides a wide range
of algorithms for classification, regression, clustering, and dimensionality reduction. Scikit-learn is well-
integrated with other Python libraries such as NumPy, SciPy, and matplotlib, making it a comprehensive
tool for machine learning tasks. It is known for its ease of use, efficient performance, and
documentation.

2. What is model deployment in the context of machine learning?

Model deployment is the process of making a trained machine learning model available for use in real-
world applications. Once a model is trained and validated, deployment involves integrating the model
into a production system where it can process new data and generate predictions. This often involves
packaging the model, creating APIs or services, and ensuring the system is scalable, reliable, and secure
for handling live data.

3. Write short notes on Spark & Spark ML.

• Apache Spark: A unified analytics engine for Big Data processing, with built-in modules for
streaming, SQL, machine learning, and graph processing. It is designed for speed and ease of
use, offering in-memory processing for faster computations compared to Hadoop MapReduce.

• Spark MLlib: Spark's machine learning library. It provides scalable algorithms for classification,
regression, clustering, and collaborative filtering. Spark MLlib also includes tools for feature
extraction, model evaluation, and hyperparameter tuning, enabling end-to-end machine
learning workflows.

4. Compare TensorFlow and PyTorch in terms of machine learning.

• TensorFlow: Developed by Google, TensorFlow is a comprehensive open-source framework for

machine learning and deep learning. It offers a static computational graph for defining models,
which can be more efficient for deployment but less flexible for research.

• PyTorch: Developed by Facebook, PyTorch is known for its dynamic computation graph, which
makes it easier to debug and experiment with. PyTorch is widely used in research and has gained
popularity for its flexibility and ease of use in building deep learning models.

Shruti Dutt 21100BTCSE09987

5. What is H2O, and how does it differ from other machine learning platforms?

H2O is an open-source machine learning platform designed for high-performance data analysis. It
supports a wide variety of machine learning algorithms such as generalized linear models, gradient
boosting machines, and deep learning.

Differences:

• Scalability: H2O is optimized for distributed computing, which enables it to scale to large
datasets.

• AutoML: H2O offers automated machine learning features (AutoML) that automatically perform
tasks like feature selection, model selection, and hyperparameter tuning.

• Integration: H2O integrates easily with Python, R, and other big data tools like Hadoop and
Spark.

6. What is Azure Machine Learning, and how does it integrate with the broader Azure ecosystem?

Azure Machine Learning (Azure ML) is a cloud-based service provided by Microsoft that enables data
scientists and developers to build, deploy, and manage machine learning models. It integrates seamlessly
with the broader Azure ecosystem, including Azure Data Lake, Azure Databricks, and Azure Kubernetes
Service, enabling end-to-end machine learning workflows, from data preparation to model deployment.

7. Describe three main components of the H2O platform and their roles.

• H2O-3: The core open-source machine learning platform providing a wide range of algorithms
for classification, regression, and clustering.

• H2O AutoML: Automates the machine learning process, helping to automatically select models,
tune hyperparameters, and evaluate results.

• H2O Driverless AI: A commercial offering focused on automated deep learning and machine
learning, emphasizing feature engineering and model interpretability.

Shruti Dutt 21100BTCSE09987

 Assignment 5

1. What is a Hadoop cluster and how do you maintain it?

A Hadoop cluster is a collection of machines that work together to store and process large datasets using
Hadoop's distributed architecture. It typically includes:

• NameNode (master node): Manages the

filesystem metadata.

• DataNodes (worker nodes): Store the actual data blocks.

• ResourceManager: Manages the distribution of computational resources.

Maintenance involves monitoring system health, ensuring optimal disk space, managing node failures,
performing software upgrades, and optimizing data storage and processing.

2. Write a short note on Apache log viewer analysis.

Apache log analysis involves inspecting and interpreting logs generated by the Apache HTTP Server. The
ELK stack (Elasticsearch, Logstash, Kibana) is a popular toolset for visualizing and analyzing Apache logs,
helping detect traffic patterns, security breaches, and performance bottlenecks.

3. What do you mean by social network mining? Write its applications.

Social network mining refers to extracting useful patterns and relationships from social network data. It
can be used to analyze user interactions, detect communities, and identify influential individuals.
Applications include:

• Sentiment analysis on social media.

• Detecting communities within networks.

• Recommending products or services.

• Detecting fraudulent activities.

Shruti Dutt 21100BTCSE09987

4. Explain the term "Social networks as a Graph." Write types of social networks.

Social networks are represented as graphs where individuals are nodes and their relationships are edges.
Types of social networks include:

• Social media networks: Facebook, Twitter.

• Professional networks: LinkedIn, GitHub.

• Communication networks: Email, messaging platforms.

5. What is the primary goal of Market Basket Analysis, and how can businesses benefit from it?

The goal of Market Basket Analysis is to identify relationships between items purchased together. It
helps businesses optimize product placement, create targeted marketing campaigns, and increase cross-
selling and up-selling opportunities.

6. What are some common challenges faced in Market Basket Analysis, particularly with large or
sparse datasets?

Challenges include data sparsity, scalability issues, overfitting of patterns, and noise in the data. These
challenges can lead to inefficient analysis and inaccurate insights.

7. What are some key performance metrics to monitor in a big data project, and why are they
important?

Key metrics include:

• Throughput: Indicates the efficiency of data processing.

• Latency: Measures the response time for individual data pieces.

• Scalability: Determines the system's ability to handle increased data volume.

• Fault Tolerance: Ensures system reliability during failures.

• Resource Utilization: Measures CPU, memory, and network efficiency.

These metrics are vital for ensuring a high-performing, scalable, and reliable Big Data system.

Shruti Dutt 21100BTCSE09987