BIG DATA IMP(21CS71)

THESE ANSWERS FROM NOTES

MODULE 1
DOWNLOAD
1 How is Data Architecture layers used for analytics? Explain with functions of
each layer.(10 Marks)

Data architecture layers play a crucial role in organizing and managing the flow of data
for analytics. The architecture is typically structured into five logical layers, each serving
specific functions that contribute to the overall analytics process. Here’s a breakdown of
each layer and its functions:

1. Layer 1: Identification of Data Sources (L1)

- Function : This layer focuses on identifying where the data will come from. Data
sources can be external (like social media, public databases) or internal (like company
databases, CRM systems). Understanding the sources is essential for effective data
ingestion and ensures that the right data is captured for analysis.

2. Layer 2: Data Ingestion (L2)

- Function : In this layer, data is absorbed into the system. This process can occur in
real-time or in batches, similar to how nutrients are absorbed by the body. The goal is to
obtain and import data for immediate use, which is crucial for timely analytics. This layer
prepares the data for further processing.
JOIN WHATSAPP CHANNEL
OR GROUP
3. Layer 3: Data Storage (L3)
- Function : Once data is ingested, it needs to be stored efficiently. This layer involves
storing data in various formats, whether on files, servers, clusters, or in the cloud. Proper
storage solutions are vital for ensuring that data is accessible and can be retrieved quickly
for analysis.

4. Layer 4: Data Processing (L4)

- Function : This layer is where the actual data processing occurs. It utilizes software
tools like MapReduce, Hive, Pig, or Spark to analyze the data. Processing can involve
cleaning, transforming, and aggregating data to prepare it for analysis. This step is critical
for ensuring that the data is in a usable format for generating insights.

5. Layer 5: Data Consumption (L5)

- Function : The final layer is where the processed data is consumed for various
applications. This includes analytics, visualizations, reporting, and exporting data to cloud
or web servers. The insights generated from this layer can drive business intelligence,
support decision-making, and enhance overall organizational performance.

1. Why is Data quality important in discover in new knowledge and decision

making?

Data quality is crucial in discovering new knowledge and making informed decisions for
several reasons:

1. Accuracy : High-quality data ensures that the information used for analysis is
accurate and reflects the true state of affairs. Inaccurate data can lead to misleading
conclusions, which can adversely affect decision-making processes.

2. Relevancy : Data must be relevant to the specific questions or problems being

addressed. High-quality data is pertinent and directly applicable, allowing organizations
to focus on the most significant insights.

3. Consistency : Maintaining data integrity over its usable life is essential. Consistent
data helps in building reliable models and analyses, which are foundational for sound
decision-making.

4. Completeness : High-quality data should be complete, meaning it should not have

missing values that could skew results. Incomplete data can lead to gaps in understanding
and hinder the discovery of new knowledge.

5. Timeliness : Data must be up-to-date to be useful. Outdated information can lead to

decisions based on past conditions that may no longer be relevant.

JOIN WHATSAPP CHANNEL

OR GROUP
6. Robustness : Quality data should be robust enough to withstand various analytical
processes without losing its integrity. This robustness ensures that the insights derived are
reliable and actionable.

7. Reduction of Noise and Outliers : High-quality data minimizes noise (irrelevant or

misleading information) and outliers (data points that deviate significantly from the
norm), which can distort analysis and lead to incorrect conclusions.

2. Explain the classification of Big data. List and explain different data sources.

Big Data can be classified based on various criteria, primarily focusing on data sources
and data formats. Here’s a detailed explanation of the classification of Big Data along with
different data sources:

Classification of Big Data

1. Data Sources : This classification is based on where the data originates. It can be
divided into:
- Traditional Data Sources : These include structured data from relational databases (
RDBMS), in-memory data tables, and data warehouses. Examples are:
- Records : Standardized data entries in databases.
- RDBMS : Traditional databases that store structured data.
- Distributed Databases : Databases that are spread across multiple locations.
- Non-Traditional Data Sources : This includes data that is generated from various
modern sources, such as:
- Machine-Generated Data : Data produced by machines, sensors, or automated
systems. For example, data from IoT devices or logs from servers.
- Human-Sourced Data : Data generated by human interactions, such as social media
posts, emails, and biometric data.
- Business Process Data : Data generated from business operations, like transaction
records and customer interactions.
- Business Intelligence Data : Data used for analysis and reporting to support
business decisions.

2. Data Formats : This classification is based on the structure of the data. It can be
categorized into:
- Structured Data : Highly organized data that fits into predefined models, such as
databases and spreadsheets.
- Semi-Structured Data : Data that does not conform to a strict structure but still
contains tags or markers to separate elements, such as XML or JSON files.
- Unstructured Data : Data that lacks a predefined format, such as text documents,
images, videos, and social media content.

Different Data Sources

JOIN WHATSAPP CHANNEL

OR GROUP
1. Operational Data Store (ODS) : A centralized database that provides a snapshot of the
current operational data from various sources, often used for reporting and analysis.

2. Data Marts : Subsets of data warehouses that focus on specific business areas or
departments, allowing for more targeted analysis.

3. Data Warehouse : A large storage system that aggregates data from different sources
for analysis and reporting, typically structured for complex queries.

4. NoSQL Databases : These databases, such as MongoDB and Cassandra, are designed
to handle unstructured and semi-structured data, providing flexibility in data storage and
retrieval.

5. Sensor Data : Data collected from various sensors, such as temperature, humidity, and
motion sensors, often used in IoT applications.

6. Audit Trail of Financial Transactions : Records that track the history of

transactions, providing insights into financial activities and compliance.

7. External Data : This includes data from outside the organization, such as web data,
social media interactions, weather data, and health records, which can enrich internal
datasets.

3. Write a short note on data storage and analysis.

Data storage and analysis are critical components of Big Data management, especially
given the vast amounts of information generated today. With the evolution of technology,
traditional storage methods have become inadequate for handling the sheer volume,
variety, and velocity of data.

Data Storage : Modern data storage solutions include distributed file systems,
operational data stores (ODS), data marts, and data warehouses. Technologies like
NoSQL databases (such as MongoDB and Cassandra) are increasingly popular due to
their ability to handle unstructured, semi-structured, and multi-structured data. Cloud
computing has also revolutionized data storage, providing scalable and elastic platforms
that can grow with the data needs of organizations. This flexibility allows businesses to
store massive datasets efficiently while ensuring quick access and retrieval.

Data Analysis : The analysis of Big Data involves using advanced tools and techniques
to extract meaningful insights from large datasets. This process typically includes data
ingestion, pre-processing, and transformation, which prepare the data for analysis. Various
software tools, such as MapReduce, Hive, Pig, and Spark, are employed for processing
the data. The goal is to identify patterns, trends, and correlations that can inform decision-
making and enhance business intelligence. Effective data analysis can
JOIN WHATSAPP CHANNEL
OR GROUP
lead to improved risk management, contextual marketing, and real-time analytics,
ultimately driving business success.

MODULE 2
DOWNLOAD
1. List Hadoop core components and plain with appropriate diagram.

The core components of the Hadoop framework are as follows:

1. Hadoop Common : This module contains the libraries and utilities required by
other Hadoop modules. It includes components for distributed file systems, general
input/output, serialization, Java RPC (Remote Procedure Call), and file-based data
structures.

2. Hadoop Distributed File System (HDFS) : A Java-based distributed file system

designed to store all types of data across clusters of computers. It provides high
throughput access to application data.

3. MapReduce : This is the programming model used in Hadoop for processing large
datasets in parallel and in batches. It consists of two main functions: the Mapper, which
processes input data, and the Reducer, which aggregates the results.

JOIN WHATSAPP CHANNEL

OR GROUP
4. YARN (Yet Another Resource Negotiator) : This component manages resources and
scheduling for applications running on the Hadoop cluster. It allows multiple data
processing engines to handle data stored in a single platform, enabling efficient resource
management.

These components work together to provide a robust framework for distributed data
processing, making Hadoop a powerful tool for handling big data.

2. Explain the working of the Hadoop Map Reduce frame

(10 Marks)

The Hadoop MapReduce framework is a powerful model for processing large datasets in a
distributed computing environment. It operates through two main functions: distributing
jobs across various nodes in a cluster and organizing the results from these nodes into a
cohesive output.

Here's a detailed breakdown of how it works:

1. Job Submission : A client node submits a request for a job to the JobTracker, which is
a daemon (background program) in the Hadoop framework. This request includes the
application task or user query that needs to be processed.

2. Resource Estimation : The JobTracker first estimates the resources required for
processing the request. This involves analyzing the current state of the slave nodes (
DataNodes) in the cluster to determine their availability and capacity.

3. Task Queueing : After assessing the resources, the JobTracker places the mapping
tasks in a queue. This ensures that tasks are organized and can be executed efficiently.

4. Task Execution : The execution of the job is managed by two main processes:
- Mapper : The Mapper takes the input data and processes it into key/value pairs. It
runs on the nodes where the data is stored, which optimizes data locality and reduces
network congestion.
- Reducer : After the Mapper completes its task, the Reducer takes the output from the
Mapper as input and combines the data tuples into a smaller set of tuples. This step is
crucial for aggregating results.

5. Monitoring and Recovery : The JobTracker continuously monitors the progress of the
tasks. If a task fails, it can restart the task on another available slot, ensuring that the job
completes successfully.

6. Data Serialization : Once the Reducer finishes processing, the output is serialized
using AVRO (a data serialization system) and sent back to the client node.

JOIN WHATSAPP CHANNEL

OR GROUP
7. Final Output : The final result is collected from the Reducer and returned to the client
, completing the job execution process.

The MapReduce programming model is designed to handle both structured and

unstructured data stored in the Hadoop Distributed File System (HDFS). It allows
developers to write programs in various languages, including Java, Python, and C++,
making it versatile for different applications.

3. How does the Hadoop MapReduce data flow work for a word count program?
Give an example. (10 Marks)

The Hadoop MapReduce data flow for a word count program follows a structured process
that involves two main phases: the Map phase and the Reduce phase. Let’s break down
how this works, using a word count example.

1. Map Phase:
In the Map phase, the input data (which could be a text file containing a large number of
words) is processed by the Mapper function. The Mapper reads the input data and
processes it line by line. For a word count program, the Mapper performs the following
steps:

- Input Splitting: The input file is split into smaller chunks (input splits), which are
processed in parallel by different Mapper tasks.
- Mapping Function: Each Mapper takes a line of text and breaks it down into
individual words. For each word, it emits a key-value pair where the key is the word itself
and the value is the number 1. For example, if the input line is "hello world hello", the
Mapper would output:
```
(hello, 1)
(world, 1)
(hello, 1)
```

2. Shuffle and Sort Phase:

After the Map phase, the output from all Mappers is shuffled and sorted. This means that
all the key-value pairs with the same key (word) are grouped together. For our example,
the output from all Mappers might look like this after shuffling:
```
(hello, 1)
(hello, 1)
(world, 1)
```

JOIN WHATSAPP CHANNEL

OR GROUP
 3. Reduce Phase:
In the Reduce phase, the Reducer takes the grouped key-value pairs and processes them.
The Reducer receives each unique word along with a list of counts. It sums up the counts
for each word. For our example, the Reducer would perform the following:

- Reducing Function: For the key "hello", it would receive the values [1, 1] and sum
them to produce (hello, 2). For "world", it would receive [1] and produce (world, 1). The
final output from the Reducer would be:
```
(hello, 2)
(world, 1)
```

This structured flow allows Hadoop to efficiently process large datasets in parallel,
making it a powerful tool for big data analytics. The entire process is managed by the
Hadoop framework, which handles job distribution, resource management, and fault
tolerance, ensuring that the word count program runs smoothly across a cluster of
machines.

4. What is APACHE Flume? Describe the feature components and working of

apache flume.

Apache Flume is a distributed, reliable, and available service designed to efficiently

collect, aggregate, and transfer large amounts of streaming data into the Hadoop
Distributed File System (HDFS). It is particularly useful for handling data from various
sources such as log files, social media messages, network traffic, sensor data, and emails.

Key Features of Apache Flume:

1. Robustness and Fault Tolerance : Flume is designed to ensure reliable data transfer,
providing mechanisms for recovery in case of failures.
2. Scalability : It can handle large volumes of data and can be scaled by adding more
Flume agents as needed.
3. Data Transfer : Flume efficiently uploads large files into Hadoop clusters, making it
suitable for big data applications.

Core Components of Apache Flume:

Apache Flume consists of three main components that work together to facilitate data
flow:

JOIN WHATSAPP CHANNEL

OR GROUP
1. Source : This component is responsible for receiving data from various sources, such
as servers or applications. A source can send data to multiple channels simultaneously.

2. Channel : A channel acts as a queue that temporarily holds the data received from the
source before it is sent to the sink. The data in the channel remains until it is consumed by
the sink. By default, data is stored in memory, but it can also be configured to be stored
on disk to prevent data loss during network failures.

3. Sink : The sink component is responsible for delivering the data to its final
destination, which could be HDFS, a local file, or another Flume agent. Each sink can
only take data from a single channel, but a Flume agent can have multiple sinks.

Working of Apache Flume:

The operation of Apache Flume can be visualized as a pipeline where data flows from the
source to the sink through the channel. When data is generated, it is captured by the
source, queued in the channel, and then processed by the sink for storage or further
analysis. This architecture allows for efficient data collection and transfer, ensuring that
large volumes of streaming data can be handled effectively.

MODULE 3
DOWNLOAD
1. Define NOSQL Explain Big Data NOSOL or Not - only SQL with its features ,
transactions and solutions.

JOIN WHATSAPP CHANNEL

OR GROUP
NoSQL, which stands for "Not Only SQL," refers to a class of non-relational data storage
systems that provide flexible data models and are designed to handle large volumes of
data, particularly in the context of big data applications. Unlike traditional SQL databases,
NoSQL databases do not require a fixed schema, allowing for more dynamic and
adaptable data structures. This flexibility is particularly beneficial for applications that
deal with semi-structured or unstructured data.

Features of NoSQL:
1. Schema Flexibility : NoSQL databases allow for schema-less data storage, meaning
data can be inserted without a predefined structure. This is ideal for applications where
data formats may change over time.

2. Horizontal Scalability : NoSQL systems are designed to scale out by adding more
servers (data nodes) to handle increased loads, making them suitable for big data
applications that require scalable storage solutions for terabytes and petabytes of data.

3. Replication and Fault Tolerance : NoSQL databases support data replication across
multiple nodes, ensuring high availability and reliability. If one node fails, others can
continue to serve requests, enhancing fault tolerance.

4. Distributable Architecture : NoSQL solutions allow for sharding, which means data
can be partitioned and distributed across multiple clusters, improving performance and
throughput.

5. Support for Various Data Models : NoSQL encompasses various data storage
models, including key-value stores, document stores (like MongoDB), column-family
stores (like Cassandra), and graph databases, each suited for different types of
applications.

Transactions in NoSQL:
NoSQL databases often sacrifice some of the traditional ACID (Atomicity, Consistency,
Isolation, Durability) properties found in SQL databases in favor of the CAP
(Consistency, Availability, Partition Tolerance) theorem and BASE (Basically Available,
Soft state, Eventually consistent) properties. This means that while NoSQL databases may
not guarantee immediate consistency, they are designed to be highly available and
partition-tolerant, making them suitable for distributed environments.

Solutions for Big Data with NoSQL:

NoSQL databases are particularly effective for managing big data due to their ability to
handle large volumes of data with high speed and efficiency. They support:
- Sparse Data Handling : NoSQL databases can efficiently manage sparse data structures
, which is common in big data scenarios.
- High Processing Speed : They are optimized for high-speed data processing, making
them suitable for real-time analytics and applications that require quick data retrieval.

JOIN WHATSAPP CHANNEL

OR GROUP
- Cost-Effectiveness : Many NoSQL solutions are open-source and can be deployed
on inexpensive hardware, reducing the overall cost of data management compared to
traditional RDBMS systems.

2. Describe graph database characteristic, typical used and examples.

Graph databases are specialized systems designed to model and manage data that is
interconnected. Here are the key characteristics, typical uses, and examples of graph
databases:

Characteristics of Graph Databases:

1. Specialized Query Languages : Graph databases often use unique query languages
tailored for their structure. For instance, RDF (Resource Description Framework) utilizes
SPARQL for querying.
2. Unique Data Modeling : They model data differently than traditional key-value,
document, or columnar stores. Instead of tables, they use nodes and edges to represent
entities and their relationships.
3. Hyper-edges : Graph databases can include hyper-edges, which allow a single edge to
connect multiple vertices, enabling more complex relationships.
4. Small Data Size Records : They consist of small records that can represent complex
interactions between graph nodes and hypergraph nodes.

Typical Uses of Graph Databases:

- Link Analysis : Analyzing connections and relationships within data, such as social
networks or web links.
- Friend of Friend Queries : Finding connections between users in social networks. -
Rules and Inference : Running queries on complex structures like class libraries or
taxonomies.
- Rule Induction : Deriving new rules from existing data patterns.
JOIN WHATSAPP CHANNEL
OR GROUP
- Pattern Matching : Identifying specific patterns within interconnected data.

Examples of Graph Databases:

- Neo4J : One of the most popular graph databases, known for its robust features and
community support.
- AllegroGraph : A graph database that supports RDF and SPARQL, often used for
semantic graph applications.
- HyperGraph : A database that allows for hyper-edges and complex relationships.
- Infinite Graph : Designed for large-scale graph processing.
- Titan : A scalable graph database optimized for storing and querying large graphs. -
FlockDB : Developed by Twitter, it is designed for managing large-scale social graphs
.

3. Compare and contrast RDBMS and Mongo DB databases.

When comparing RDBMS (Relational Database Management Systems) and MongoDB,

there are several key differences and similarities to consider:

1. Data Structure :
- RDBMS : Uses a structured format with tables, rows, and columns. Each table has a
predefined schema, meaning the structure of the data must be defined before data can be
inserted.
- MongoDB : Utilizes a schema-less design, storing data in collections of documents.
Each document can have a different structure, allowing for greater flexibility in data
storage.

2. Data Storage :
- RDBMS : Data is stored in tables, where each row represents a record and each
column represents a field in that record.
- MongoDB : Data is stored in collections, which are analogous to tables, but each
document (similar to a row) can contain varying fields and data types.

3. Relationships :
- RDBMS : Supports complex relationships through the use of joins between tables. -
MongoDB : Does not use joins; instead, it supports embedded documents,
allowing related data to be stored together within a single document.

4. Primary Key :
- RDBMS : Each table has a primary key that uniquely identifies each record.
- MongoDB : Each document has a default primary key (id) provided by MongoDB
itself.

5. Replication and Availability :

JOIN WHATSAPP CHANNEL

OR GROUP
 - RDBMS : Typically requires manual setup for replication and may not be as fault-
tolerant.
- MongoDB : Uses replica sets for high availability, where multiple copies of data are
stored across different servers, ensuring fault tolerance.

6. Query Language :
- RDBMS : Uses SQL (Structured Query Language) for querying data, which is
powerful but requires a structured approach.
- MongoDB : Uses a document-based query language that allows for dynamic queries,
which can be nearly as powerful as SQL but is designed for the document model.

7. Performance and Scalability :

- RDBMS : Scaling can be challenging, often requiring vertical scaling (upgrading
existing hardware).
- MongoDB : Designed for horizontal scalability, allowing data to be sharded across
multiple servers, which enhances performance and throughput.

8. Flexibility :
- RDBMS : Rigid structure due to predefined schemas, making it less adaptable to
changes in data requirements.
- MongoDB : Highly flexible, allowing for easy modifications to the data structure
without downtime.

4. What are the different ways of handling Big Data Problems?

There are several effective ways to handle Big Data problems, each designed to optimize
performance and manage large datasets efficiently. Here are some key strategies:

JOIN WHATSAPP CHANNEL

OR GROUP
1. Even Distribution of Data : One approach is to evenly distribute the data across
clusters using a method called consistent hashing. This involves generating pointers for a
collection using a hashing algorithm, which helps in determining the data location within
the cluster. The hash ring serves as a map of hashes with locations, allowing for
consistent assignment and usage of datasets to specific processors.

2. Replication for Read Requests : Replication involves creating real-time backup

copies of data across multiple nodes in a cluster. This strategy enhances data availability
and reliability, allowing for horizontal distribution of client read requests. By having
multiple copies, the system can ensure that data retrieval remains efficient even in the
event of node failures.

3. Moving Queries to Data Nodes : Instead of sending queries to the nodes, many
NoSQL data stores move the queries to the data itself. This approach is more efficient and
is a requirement in Big Data solutions, as it reduces the amount of data transferred over
the network and speeds up query processing.

4. Query Distribution : Distributing client queries across multiple nodes is another

effective method. This involves analyzing client queries at the analyzers, which then
evenly distribute the queries to data nodes or replica nodes. This parallel processing of
queries significantly enhances performance and throughput.

5. Sharding : Sharding is the practice of dividing a large database into smaller, more
manageable pieces called shards, which are distributed across different servers or clusters.
This not only improves performance but also allows for horizontal scalability as more
machines can be added to handle increased data loads.

5. Explain different components of Cassandra (ii) Explain different data types

built into Cassandra.

JOIN WHATSAPP CHANNEL

OR GROUP
Cassandra is a powerful NoSQL database designed to handle large amounts of data across
many servers, providing high availability with no single point of failure. Let's break down
the components and data types built into Cassandra:

(i) Components of Cassandra:

1. Cluster : A cluster is made up of multiple nodes, which are the individual servers that
store data. Each cluster can contain multiple keyspaces.

2. Keyspace : This is a namespace that groups together multiple column families. It

defines the replication strategy for the data within it. Typically, there is one keyspace per
application.

3. Column : A column in Cassandra consists of three parts: a column name, a value, and
a timestamp. This structure allows for efficient storage and retrieval of data.

4. Column Family : This is a collection of columns that are grouped together by a row
key. It is similar to a table in relational databases but is more flexible in terms of schema.

Cassandra manages keyspaces by partitioning keys into ranges and assigning these ranges
to specific nodes, which helps in distributing the data evenly across the cluster.

(ii) Data Types Built into Cassandra:

Cassandra supports a variety of data types to accommodate different kinds of data. Here
are some of the key data types:

1. Primitive Types : These include basic data types such as:

- `int`: Integer values.
- `text`: String values.
- `boolean`: True or false values.
- `float`: Floating-point numbers.
- `double`: Double-precision floating-point numbers.
- `timestamp`: Date and time values.

2. Collection Types : These allow for the storage of multiple values in a single column:

- List : An ordered collection of elements.

- Set : An unordered collection of unique elements.
- Map : A collection of key-value pairs.

3. User-Defined Types (UDTs) : Cassandra allows users to define their own data types,
which can encapsulate multiple fields of different types, providing a way to model
complex data structures.

JOIN WHATSAPP CHANNEL

OR GROUP
4. Tuple : A fixed-length collection of elements, which can be of different types.

5. Blob : A binary large object that can store any type of binary data.

6. Describe different CQL commands and their functionalities.(ii) Write a short

note on NOSQL to Manage Big Data.

(i) Cassandra Query Language (CQL) commands are essential for interacting with
Cassandra databases. Here are some key commands and their functionalities:

1. DESCRIBE CLUSTER : This command provides a description of the cluster,

including its configuration and status.
2. DESCRIBE SCHEMA : It displays the schema of the database, detailing the
structure of the data.
3. DESCRIBE KEYSPACES : This command lists all the keyspaces in the cluster,
which are namespaces that group related column families.
4. DESCRIBE KEYSPACE «keyspace name» : It gives detailed information about a
specific keyspace, including its associated column families.
5. DESCRIBE TABLES : This command lists all the tables within the current keyspace
.
6. DESCRIBE TABLE «table name» : It provides the schema of a specific table,
detailing its columns and data types.
7. DESCRIBE INDEX «index name» : This command describes a specific index,
including its properties and the table it is associated with.
8. DESCRIBE MATERIALIZED VIEW «view name» : It gives details about a
materialized view, which is a pre-computed query result stored as a table.
9. DESCRIBE TYPES : This command lists user-defined types in the keyspace.
10. DESCRIBE TYPE «type name» : It provides details about a specific user-defined
type.
11. DESCRIBE FUNCTIONS : This command lists all user-defined functions in the
keyspace.
12. DESCRIBE FUNCTION «function name» : It describes a specific user-defined
function.
13. DESCRIBE AGGREGATES : This command lists user-defined aggregate functions
in the keyspace.

(ii) NoSQL databases play a crucial role in managing Big Data due to their unique
characteristics and capabilities. They are designed to handle large volumes of data that
traditional SQL databases struggle with. Here are some key points about using NoSQL to
manage Big Data:

1. Scalability : NoSQL databases are built for horizontal scalability, allowing them to
expand by adding more servers to handle increased data loads efficiently. This is
essential for managing terabytes and petabytes of data.
JOIN WHATSAPP CHANNEL
OR GROUP
2. Flexibility : NoSQL databases support schema-less data models, meaning data can
be inserted without a predefined structure. This flexibility is vital for Big Data
applications where data formats can vary widely.

3. High Availability : NoSQL solutions often use data replication across multiple nodes
, ensuring that data remains accessible even if some nodes fail. This fault tolerance is
critical for maintaining uptime in Big Data environments.

4. Performance : NoSQL databases are optimized for high-speed data processing and
can handle a large number of read and write operations simultaneously, making them
suitable for real-time analytics.

5. Distributed Architecture : NoSQL databases utilize a distributed architecture,

allowing data to be stored across multiple servers or clusters. This distribution enhances
performance and throughput, making it easier to manage large datasets.

6. Support for CAP and BASE : Unlike traditional databases that adhere strictly to
ACID properties, NoSQL databases often follow the CAP theorem (Consistency,
Availability, Partition tolerance) and BASE (Basically Available, Soft state, Eventually
consistent) principles, which are more suited for distributed systems handling Big Data.

MODULE 4
DOWNLOAD

1. Hive Built - in functions.

Hive provides a variety of built-in functions that can be used for data manipulation and
analysis. Here are some of the key built-in functions along with their return types, syntax,
and descriptions:

1. Count Function
- Return Type: BIGINT
- Syntax: `count( )`, `count(expr)`
- Description: Returns the total number of retrieved rows.

2. Sum Function
- Return Type: DOUBLE
- Syntax: `sum(col)`, `sum(DISTINCT col)`
- Description: Returns the sum of the elements in the group or the sum of the distinct
values of the column in the group.

3. Average Function
- Return Type: DOUBLE
JOIN WHATSAPP CHANNEL
OR GROUP
 - Syntax: `avg(col)`, `avg(DISTINCT col)`
- Description: Returns the average of the elements in the group or the average of the
distinct values of the column in the group.

4. Minimum Function
- Return Type: DOUBLE
- Syntax: `min(col)`
- Description: Returns the minimum value of the column in the group.

5. Maximum Function
- Return Type: DOUBLE
- Syntax: `max(col)`
- Description: Returns the maximum value of the column in the group.

These functions are similar in usage to SQL aggregate functions and are essential for
performing data analysis within Hive.

2. Explain with respect to Hive QI

i)Hive QL Data Definition Language (DDL). ii) Hive QL Data Manipulation Language
(DML).

Sure! Let's break down HiveQL into its two main components: Data Definition Language
(DDL) and Data Manipulation Language (DML).

i) HiveQL Data Definition Language (DDL)

DDL in HiveQL is used to define and manage the structure of the database and its tables.
Here are some key commands:

- CREATE DATABASE : This command creates a new database. For example:

```sql
CREATE DATABASE toys_companyDB;
```

- SHOW DATABASES : This command lists all the databases available in Hive.

- CREATE TABLE : This command creates a new table within a database. You can
specify the schema (columns and their data types) and other properties. For example:
```sql
CREATE TABLE toys_tbl (
puzzle_code STRING,
pieces SMALLINT,
cost FLOAT
);
```

JOIN WHATSAPP CHANNEL

OR GROUP
- CREATE SCHEMA : Similar to creating a database, this command is used to create a
schema.

- DROP TABLE : This command removes a table from the database.

- ALTER TABLE : This command modifies an existing table structure, such as adding
or dropping columns.

ii) HiveQL Data Manipulation Language (DML)

DML in HiveQL is used for managing the data within the tables. Here are some important
commands:

- LOAD DATA : This command is used to load data into a table from a specified file
path. For example:
```sql
LOAD DATA LOCAL INPATH '<file path>' INTO TABLE <table name>;
```

- SELECT : This command retrieves data from one or more tables. You can use various
clauses like WHERE, GROUP BY, and ORDER BY to filter and organize the results.
For example:
```sql
SELECT FROM toys_tbl WHERE cost > 10;
```

- DROP TABLE : This command not only removes the table structure but also the data
contained within it.

- INSERT : While Hive does not support traditional row-level updates and deletes, you
can insert new data into a table.

3. Explain feature and applications of PIG.(10 Marks)

Apache Pig is a high-level platform for creating programs that run on Apache Hadoop. It
simplifies the complexities of writing MapReduce programs by providing a dataflow
language known as Pig Latin. Let's break down its architecture, features, and applications
in detail.

Features of Apache Pig

JOIN WHATSAPP CHANNEL

OR GROUP
1. High-Level Language : Pig Latin is designed to be user-friendly, allowing
programmers to write complex data transformations without needing to delve into Java
code. It resembles SQL, making it accessible to those familiar with database querying.

2. Rich Set of Operators : Pig Latin includes a variety of built-in operators such as `
group`, `join`, `filter`, `limit`, `order by`, `parallel`, `sort`, and `split`, which facilitate
various data manipulation tasks.

3. User Defined Functions (UDFs) : Users can create custom functions that extend Pig's
capabilities, allowing for specialized processing that may not be covered by built-in
functions.

4. Support for Various Data Types : Pig can handle structured, semi-structured, and
unstructured data, making it versatile for different data sources.

5. Multi-Query Approach : Pig allows for the execution of multiple queries in a single
script, significantly reducing the amount of code needed compared to traditional
MapReduce.

6. Interactive Shell (Grunt) : Pig provides an interactive shell called Grunt, where users
can execute Pig Latin commands and scripts in real-time.

7. Automatic Optimization : Before execution, Pig automatically optimizes the tasks,

which can lead to improved performance and resource utilization.

Applications of Apache Pig

1. Analyzing Large Datasets : Pig is widely used for processing and analyzing large
volumes of data, making it suitable for big data applications.

2. Ad-Hoc Processing : It excels in executing tasks that require ad-hoc processing,

allowing users to quickly analyze data without extensive setup.

3. Web Log Processing : Pig is effective for processing large data sources such as web
logs, enabling organizations to derive insights from user interactions.

4. Streaming Data Analysis : It can handle streaming online data, making it useful for
real-time analytics and monitoring.

5. Data Processing for Search Platforms : Pig is often employed in search engine data
processing, where it can analyze and index large datasets efficiently.

6. Time-Sensitive Data Loads : Pig can process time-sensitive data quickly, such as
analyzing Twitter data to identify user behavior patterns and generate recommendations.

JOIN WHATSAPP CHANNEL

OR GROUP
4. Describe the significance of apache pig in hadoop. (10 Marks)

Apache Pig plays a crucial role in the Hadoop ecosystem, particularly in simplifying the
process of handling large datasets. Here are several key points that highlight its
significance:

1. Abstraction Over MapReduce : Pig serves as an abstraction layer over the complex
MapReduce programming model. This means that it allows users to write data processing
tasks without needing to delve into the intricacies of MapReduce, which can be quite
complex and verbose.

2. High-Level Dataflow Language : Pig uses a high-level dataflow language called Pig
Latin, which is designed to be more intuitive and easier to use than Java-based
MapReduce code. Pig Latin is similar to SQL, making it accessible for users with a basic
understanding of SQL. This reduces the learning curve for data analysts and developers.

3. Simplified Data Processing : With Pig, users can perform complex data
transformations and manipulations with significantly less code. For instance, a Pig script
that might take 10 lines can replace a MapReduce job that requires around 200 lines of
Java code. This efficiency is particularly beneficial for rapid development and
prototyping.

4. Support for Various Data Types : Pig can handle structured, semi-structured, and
unstructured data, making it versatile for different data processing tasks. This capability is
essential in today's data landscape, where data comes in various formats.

5. Interactive and Batch Processing : Pig supports multiple execution modes, including
local mode for testing and MapReduce mode for processing data stored in HDFS. It also
allows for interactive script execution through the Grunt shell, as well as batch processing
by writing scripts in files with a .pig extension.

6. User Defined Functions (UDFs) : Pig allows developers to create UDFs, enabling
them to extend Pig's capabilities by writing custom functions in languages like Java. This
flexibility is vital for addressing specific processing needs that may not be covered by
built-in functions.

7. Automatic Optimization : Before execution, Pig performs automatic optimization of

tasks, which enhances performance and efficiency. This means that users can focus on
writing their data processing logic without worrying about the underlying optimization
details.

8. Data Processing for Real-Time Applications : Pig is particularly useful for

processing time-sensitive data loads, such as analyzing web logs or streaming data from

JOIN WHATSAPP CHANNEL

OR GROUP
platforms like Twitter. This capability allows organizations to derive insights and make
decisions based on real-time data.

9. Integration with Hadoop Ecosystem : As part of the Hadoop ecosystem, Pig

seamlessly integrates with HDFS, allowing it to read and write data efficiently. This
integration is crucial for leveraging the distributed storage and processing capabilities
of Hadoop.

10. Community and Support : Being an Apache project, Pig benefits from a robust
community of developers and users. This community support ensures continuous
improvement, updates, and a wealth of resources for users to tap into.

5. With a neat diagram, explain MapReduce Programming model. How does

MapReduce enables query processing quickly in Big Data Problems?(10 Marks)

The MapReduce programming model is a powerful framework designed for processing

large datasets in a distributed computing environment, particularly within the Hadoop
ecosystem. It consists of two primary tasks: the Map task and the Reduce task, which
work together to efficiently process and analyze vast amounts of data.

Map Task
1. Input Data Handling : The Map task takes an input dataset, which is typically stored
in the Hadoop Distributed File System (HDFS), and processes it in parallel across various
nodes in a cluster. Each piece of data is treated as a key-value pair.
2. Mapping Function : The `map()` function is applied to each key-value pair (k1, v1).
The key (k1) represents a unique identifier, while the value (v1) is the data associated
with that key. The output of the `map()` function can either be zero (if no relevant data is
found) or a set of intermediate key-value pairs (k2, v2).
3. Parallel Processing : By distributing the input data across multiple nodes, the Map
task allows for parallel processing, which significantly speeds up the data handling
process.

Reduce Task
1. Combining Results : After the Map tasks complete, the output (intermediate key-
value pairs) is sent to the Reduce task. The Reduce task takes these outputs as input and
combines them into a smaller, more manageable set of data.

JOIN WHATSAPP CHANNEL

OR GROUP
2. Aggregation : The `reduce()` function is responsible for aggregating the data based on
the keys produced by the Map tasks. This step is crucial for summarizing the results and
producing the final output.

Query Processing in Big Data

MapReduce enables quick query processing in Big Data problems through several
mechanisms:

1. Scalability : The framework can scale horizontally by adding more nodes to the
cluster, allowing it to handle larger datasets without a significant drop in performance.
2. Data Locality : MapReduce optimizes processing by co-locating compute and storage
nodes. This means that tasks are scheduled on nodes where the data is already present,
reducing network traffic and latency.
3. Fault Tolerance : The framework is designed to handle node failures gracefully. If a
TaskTracker fails, the JobTracker can restart the task on another node, ensuring that the
processing continues without significant delays.
4. Efficient Resource Utilization : The execution framework manages the distribution of
tasks, scheduling, and synchronization, which allows for efficient use of cluster resources
and minimizes idle time.

6. (i)Differentiate between Pig and MapReduce

- Type of Language : Pig is a high-level dataflow language, while MapReduce is a low-
level data processing paradigm. This means that Pig allows for more abstraction and
easier coding compared to the more complex and rigid structure of MapReduce.

- Ease of Use : In Pig, performing operations like joins, filters, and sorting is relatively
simple, making it accessible for users with basic SQL knowledge. In contrast, MapReduce
requires complex Java implementations, which can be challenging for those not familiar
with Java.

- Code Length : Pig uses a multi-query approach that significantly reduces the length of
code needed to perform tasks. For example, a Pig script might require only 10 lines of
code, whereas the equivalent MapReduce program could require around 200 lines.

- Compilation : Pig does not require a lengthy compilation process; it converts operators
internally into MapReduce jobs. On the other hand, MapReduce jobs involve a long
compilation process before execution.

- Data Types : Pig supports nested data types such as tuples, bags, and maps, which are
not available in MapReduce.

JOIN WHATSAPP CHANNEL

OR GROUP
 MODULE 5
DOWNLOAD

1. How does regression analysis predict the value of the dependent variable in case
of linear regression?(10 Marks)

Regression analysis, particularly linear regression, predicts the value of a dependent

variable by establishing a mathematical relationship between that variable and one or
more independent variables. Here's a detailed breakdown of how this process works:

1. Modeling the Relationship : In linear regression, we assume that there is a linear

relationship between the independent variable (predictor) and the dependent variable (
outcome). This relationship is expressed through a linear equation of the form:

2. Fitting the Regression Line : The goal of regression analysis is to find the best- fitting
line through a scatter plot of data points. This line minimizes the deviation (or error)
between the observed values and the values predicted by the model. The best-fitting line
is known as the regression line.

3. Calculating Predictions : Once the regression equation is established, you can

predict the value of the dependent variable for any given value of the independent
variable. For example, if you have a student's high school percentage, you can use
the regression equation to predict their GPA in college.

4. Understanding Errors : The difference between the observed values and the predicted
values is referred to as the error. The regression analysis aims to minimize these errors
across all data points, ensuring that the predictions are as accurate as possible.

5. Correlation and Prediction : Linear regression is closely related to correlation. A

strong correlation between the independent and dependent variables indicates that the
regression model will likely provide reliable predictions. The correlation coefficient (
denoted as \(r\)) quantifies the strength and direction of this relationship.

6. Interpreting R-squared : The R-squared value, which ranges from 0 to 1, indicates

how well the independent variable explains the variability of the dependent variable. A
higher R-squared value suggests a better fit of the model to the data, meaning that the
predictions made by the regression equation are more reliable.

2. Explain with example and algorithm, the working principle of Apriori process
for adopting the subset of frequent item sets as a frequent itemset.(10 Marks)

The Apriori algorithm is a fundamental method used in data mining for frequent itemset
mining and association rule learning. Its working principle is based on the Apriori
principle, which states that if an itemset is frequent, then all of its subsets must also be

JOIN WHATSAPP CHANNEL

OR GROUP
frequent. This property allows the algorithm to reduce the number of itemsets that need to
be examined, making the mining process more efficient.

Working Principle of the Apriori Algorithm

1. Initialization : The algorithm starts by identifying all the individual items in the
database and counting their occurrences. This forms the first set of candidate itemsets,
known as 1-itemsets.

2. Candidate Generation : For each subsequent iteration, the algorithm generates new
candidate itemsets (k+1 itemsets) from the frequent itemsets of the previous iteration (k-
itemsets). This is done by joining the frequent k-itemsets with themselves.

3. Pruning : The algorithm then prunes the candidate itemsets by removing those that do
not have any frequent subsets. This is based on the anti-monotone property of support,
which states that if an itemset is not frequent, then none of its supersets can be frequent.

4. Support Counting : The remaining candidate itemsets are then tested against the
database to count their support (the frequency of occurrence). If the support of an itemset
meets or exceeds a predefined minimum support threshold, it is considered a frequent
itemset.

5. Iteration : Steps 2 to 4 are repeated until no new frequent itemsets can be generated.

Example of the Apriori Algorithm

Let's consider a simple example with a transaction database:

| Transaction ID | Items Purchased |

|----------------|-----------------------|
|1 | {A, B, C} |
|2 | {A, B} |
|3 | {A, C} |
|4 | {B, C} |
|5 | {A, B, C} |

Step 1: Count 1-itemsets

- A: 4
- B: 4
- C: 3

Assuming the minimum support threshold is 3, all individual items (A, B, C) are frequent.

Step 2: Generate 2-itemsets -

Candidates: {A, B}, {A, C}, {B, C}
JOIN WHATSAPP CHANNEL
OR GROUP
 Step 3: Count 2-itemsets
- {A, B}: 3 (Transactions 1, 2, 5)
- {A, C}: 2 (Transactions 1, 3)
- {B, C}: 3 (Transactions 1, 4, 5)

Frequent 2-itemsets are {A, B} and {B, C} (both have support ≥ 3).

Step 4: Generate 3-itemsets

- Candidate: {A, B, C}

Step 5: Count 3-itemsets

- {A, B, C}: 2 (Transactions 1, 5)

Since {A, B, C} does not meet the minimum support threshold, the algorithm terminates.

3. Define Web Mining. Discuss the broad clássification of web mining and their
applications.(10 Marks)

Web mining refers to the use of techniques and algorithms to extract knowledge from web
data, which is available in the form of web documents and services. It encompasses a
variety of methods aimed at discovering patterns and insights from the vast amount of
information available on the internet. The primary goal of web mining is to transform raw
web data into meaningful knowledge that can be utilized for various applications.

Web mining can be broadly classified into three main categories based on the types of
web data being mined:
JOIN WHATSAPP CHANNEL
OR GROUP
1. Web Content Mining : This involves extracting useful information from the content
of web documents. The content can include text, images, audio, video, or structured
records like lists and tables. Applications of web content mining include:
- Classifying web documents into categories.
- Identifying topics of web documents.
- Finding similar web pages across different web servers.
- Enhancing query relevance through recommendations and filters.

2. Web Structure Mining : This focuses on discovering structural information from the
web. It analyzes the relationships between web pages, often represented as a graph where
web pages are nodes and hyperlinks are edges. Applications include:
- Identifying interesting graph patterns.
- Pre-processing the web graph to derive metrics like PageRank.
- Understanding the interconnections between different web resources.

3. Web Usage Mining : This involves analyzing user behavior and patterns based on
web usage data, such as server logs that record user interactions with a website.
Applications of web usage mining include:
- User identification and session creation.
- Detecting malicious activities and filtering content.
- Extracting usage path patterns to improve website design and user experience.

4. Define the term Social network. Explain social network as graphs with
Centralities, Ranking and Anomaly Detection.

A social network is a social structure composed of individuals or organizations, referred to

as "nodes," which are interconnected by various types of relationships, such as friendship,
kinship, financial exchanges, or shared beliefs. This interconnectedness allows for the
analysis of social dynamics and interactions within the network.

When we consider social networks as graphs, we can represent these nodes and their
relationships as a graph, where nodes are the entities and edges represent the connections
between them. This graphical representation enables the application of various analytical
metrics to understand the structure and behavior of the network.

Centralities
JOIN WHATSAPP CHANNEL
OR GROUP
Centrality metrics are crucial for analyzing the importance of nodes within a social
network. Key centrality measures include:

1. Degree Centrality : This metric counts the number of direct connections a node has. A
higher degree indicates a more connected node.
2. Closeness Centrality : This measures how close a node is to all other nodes in the
network, reflecting its ability to access information quickly.
3. Betweenness Centrality : This indicates how often a node acts as a bridge along the
shortest path between two other nodes, highlighting its role in facilitating communication
within the network.
4. Eigenvector Centrality : This metric considers not just the number of connections a
node has, but also the quality and influence of those connections.

Ranking
Ranking within social networks often involves determining the significance of nodes
based on their centrality scores. For instance, nodes with high betweenness centrality may
be ranked higher due to their critical role in connecting disparate parts of the network.
PageRank, a well-known algorithm originally used by Google, is another method for
ranking nodes based on their connections and the importance of those connections.

Anomaly Detection
Anomaly detection in social networks involves identifying unusual patterns or behaviors
that deviate from the norm. This can be achieved through the analysis of centrality metrics
and the structure of the network. For example, a node exhibiting an unusually high degree
of connections compared to others may indicate a spam account or a potential security
threat. Techniques such as ego-networks, which focus on a specific node and its
immediate connections, can help in identifying these anomalies.

5. What are outliers? Describe the reasons for the presence of outliers in a
relationship.(10 Marks)

Outliers are data points that significantly differ from the rest of the dataset. They can
appear as if they do not belong to the dataset and are often identified as points that are
numerically distant from other observations. The presence of outliers is important to
consider because they can affect the quality of data analysis, potentially skewing results
and leading to incorrect conclusions.

There are several reasons for the presence of outliers in a relationship:

1. Anomalous Situations : Outliers may arise from unusual or rare events that do not
reflect the typical behavior of the data.

2. Presence of Unknown Facts : Sometimes, outliers can indicate the existence of

previously unknown factors or variables that influence the data.
JOIN WHATSAPP CHANNEL
OR GROUP
3. Human Error : Errors during data entry or data collection can lead to outliers. For
instance, a typo in a numerical entry can create a data point that is far removed from the
expected range.

4. Intentional Reporting Errors : In cases where sensitive data is involved, participants

may intentionally report incorrect information, leading to outliers. This is common in self-
reported measures.

5. Sampling Error : Outliers can also occur when an unfitted or biased sample is
collected from the population, resulting in data points that do not accurately represent the
overall dataset.

Understanding and identifying outliers is crucial for improving data quality and ensuring
accurate analysis, as they can significantly influence statistical outcomes and predictions.

JOIN WHATSAPP CHANNEL

OR GROUP