Big Data Analytics

UNIT-I

1. What is Big Data and why is it important?

Answer: Big Data refers to extremely large datasets that are too complex and vast to be processed
using traditional data processing methods. It is important because it allows businesses and
organizations to analyze and gain insights from data, leading to better decision-making and
innovations.

Example: A social media platform analyzing billions of user interactions to improve user experience
and target advertising effectively.

2. What are the main challenges of conventional data systems in handling Big Data?

Answer: Conventional data systems struggle with:

 Volume: Handling massive amounts of data.

 Velocity: Processing data at high speed.
 Variety: Managing diverse data types (structured, unstructured).
 Veracity: Ensuring data accuracy and quality.

Example: A traditional database might fail to handle real-time transaction data from millions of online
shoppers during a sales event.

3. How has the nature of data evolved over time?

Answer: Data has evolved from structured, well-organized formats (like spreadsheets) to include
unstructured data (like social media posts, videos) and semi-structured data (like JSON files).

Example: Previously, businesses relied on structured data like sales records, but now they also
analyze unstructured data such as customer reviews on social media.

4. What is analytic scalability and why is it important?

Answer: Analytic scalability is the ability of a system to handle increasing amounts of data efficiently.
It's important because as data grows, the system must scale to provide timely insights without
performance degradation.

Example: An e-commerce company scaling its analytics system to handle and analyze data from
thousands of new customers during a holiday season.

5. What is intelligent data analysis?

Answer: Intelligent data analysis involves using advanced techniques and algorithms, like machine
learning, to extract meaningful patterns and insights from data automatically.
Example: A streaming service recommending movies based on a user’s viewing history using machine
learning algorithms.

6. What is the difference between analysis and reporting?

Answer:

 Analysis: Involves exploring data to uncover patterns, trends, and insights.

 Reporting: Involves presenting data in an organized format, like dashboards and reports, often
summarizing the results of analysis.

Example: Analysis might involve examining customer behavior data to find trends, while reporting
presents these findings in a monthly performance report.

7. What are some modern data analytic tools?

Answer: Modern data analytic tools include:

 Hadoop: For distributed storage and processing.

 Spark: For fast data processing.
 Tableau: For data visualization.
 Python (with libraries like pandas and scikit-learn): For data analysis and machine learning.

Example: A data scientist using Spark to process large datasets quickly and Tableau to create
interactive visualizations.

8. What are sampling distributions?

Answer: A sampling distribution is the probability distribution of a statistic (like the mean) based on a
large number of samples from a population.

Example: If you repeatedly take samples of 50 students’ test scores from a school and calculate the
mean score for each sample, the distribution of these means is the sampling distribution.

9. What is re-sampling in statistics?

Answer: Re-sampling involves repeatedly drawing samples from a dataset and calculating a statistic to
estimate the sampling distribution. Common methods include bootstrapping and permutation tests.

Example: Using bootstrapping to estimate the confidence interval for the mean height of a sample of
people by repeatedly sampling with replacement from the original dataset.

10. What is prediction error and why is it important?

Answer: Prediction error is the difference between the actual value and the predicted value. It is
important because it measures the accuracy of a predictive model.
Example: In a housing price prediction model, if a house's actual price is $300,000 and the model
predicts $320,000, the prediction error is $20,000. Lower prediction errors indicate more accurate
models.

UNIT-II

1. What are streams in the context of data?

Answer: Streams refer to continuous flows of data that are generated in real-time from various
sources. Unlike traditional batch processing, where data is collected, stored, and then processed,
stream processing handles data as it arrives, making it suitable for real-time analytics.

Example: Imagine a weather station that continuously sends temperature readings every second.
These readings form a data stream. Instead of waiting for a full day’s worth of data, we can analyze
these readings as they come in, minute by minute.

2. What is the Stream Data Model and Architecture?

Answer: The Stream Data Model represents data as a continuous sequence of elements that are
processed as they arrive. The architecture includes:

 Data Producers: Sources that generate data (e.g., sensors, social media).
 Stream Processors: Systems that process the data in real-time (e.g., filtering, aggregating).
 Data Consumers: End-users or storage systems that receive processed data.

Example: In an online shopping site, customer activities (like clicks, views, purchases) are data
producers. These activities are processed in real-time to recommend products (stream processors)
and the recommendations are shown to users instantly (data consumers).

3. What is Stream Computing?

Answer: Stream computing is the processing of continuous data streams in real-time. Instead of
storing data and analyzing it later, stream computing processes data as it arrives to perform
immediate analytics, detect patterns, or make decisions on the spot.

Example: A financial institution monitoring transactions in real-time to detect and prevent fraudulent
activities as soon as they occur, rather than after the fact.

4. What is Sampling Data in a Stream?

Answer: Sampling data in a stream means selecting a subset of data points from the continuous
stream for analysis. This helps in managing the large volume of data by focusing on a smaller,
manageable portion without losing significant insights.

Example: From a stream of sensor data, you might select every 10th reading to analyze trends in
temperature without processing every single data point.
5. What is Filtering Streams?

Answer: Filtering streams involves removing unwanted data and retaining only the relevant
information. This makes the data stream more manageable and ensures that only useful data is
processed further.

Example: In a live news feed, you might filter out all non-technology-related posts if you are only
interested in tech news. This way, you only see posts related to technology.

6. How do you count distinct elements in a stream?

Answer: Counting distinct elements in a stream means identifying and counting unique items as they
appear in the data stream. This is important for understanding the variety and diversity within the
data.

Example: A website might count the number of unique visitors by tracking each distinct IP address
that accesses the site over time, helping to understand how many different people are visiting.

7. What is meant by estimating moments in a stream?

Answer: Estimating moments in a stream involves calculating statistical properties like the mean
(average) and variance (spread) of the data as it flows in real-time. This helps in understanding the
data’s characteristics without needing to store and process it all at once.

Example: A traffic monitoring system calculating the average speed of cars on a highway in real-time
to provide up-to-date traffic information.

8. How do you count occurrences within a window in a stream?

Answer: Counting occurrences within a window means keeping track of how often an event happens
within a specific timeframe or a set number of data points. This helps in understanding the frequency
of events over time.

Example: Counting how many times a particular hashtag is used in tweets during the last hour to
measure its popularity over that specific period.

9. What is a decaying window?

Answer: A decaying window is a method that reduces the importance of older data points over time.
This means recent data has more influence on the analysis than older data, which is useful for tracking
trends that change over time.

Example: In stock market analysis, recent transactions are given more weight than older ones to
reflect the current market conditions more accurately, helping traders make better decisions based
on the latest data.

10. What are some real-time analytics platform (RTAP) applications?

Answer: Real-time analytics platforms (RTAP) are used to process and analyze data in real-time,
providing immediate insights and actions. These platforms are essential for applications that require
quick responses to changing data.

Examples:

 Real-Time Sentiment Analysis: Analyzing social media posts as they are made to gauge public
opinion about a new product, allowing companies to respond quickly to customer feedback.
 Stock Market Predictions: Processing stock price data in real-time to predict market trends
and make instant trading decisions, helping investors capitalize on market movements as they
happen.

UNIT-III

1. What is Big Data Analytics and why is it important?

Answer: Big Data Analytics is the process of examining large and diverse datasets to uncover hidden
patterns, correlations, market trends, customer preferences, and other valuable information. It is
important because it helps organizations make better decisions by providing insights that were
previously unattainable due to the sheer volume, velocity, and variety of the data.

Example: A retail company can use Big Data Analytics to analyze customer purchase history, social
media interactions, and browsing behavior. This helps them to personalize marketing campaigns,
optimize inventory management, and improve customer service, leading to increased sales and
customer loyalty.

2. How do you visualize and explore data?

Answer: Data visualization and exploration involve using graphical representations to understand
data trends, patterns, and outliers. Common techniques include bar charts, histograms, pie charts,
scatter plots, and heatmaps. These visualizations help in identifying significant insights that might not
be obvious from raw data.

Example: A business analyst uses a scatter plot to visualize the relationship between advertising
spend and sales revenue. The plot helps identify whether higher advertising spend correlates with
increased sales, which is crucial for budget planning.

3. What are the basics of R and RStudio?

Answer: R is a programming language and software environment used for statistical computing and
graphics. RStudio is an integrated development environment (IDE) for R that provides a user-friendly
interface for coding, debugging, and visualizing data.
Example: A data scientist uses RStudio to write R scripts for analyzing a dataset of customer
transactions. The IDE's features, like syntax highlighting and debugging tools, make it easier to
develop and test the analysis code.

4. How do you perform basic analysis in R?

Answer: Basic analysis in R involves data import, cleaning, and summarization. Key functions include
reading data from various sources, handling missing values, and calculating basic statistics like mean,
median, and standard deviation.

Example: An analyst imports a CSV file containing sales data using the read.csv() function in R. They
then clean the data by removing rows with missing values and calculate summary statistics to
understand the overall sales performance.

5. What are some intermediate R techniques for data analysis?

Answer: Intermediate R techniques include data manipulation using packages like dplyr, data
visualization using ggplot2, and performing statistical tests. These techniques help in more
sophisticated data analysis and visualization.

Example: Using dplyr, an analyst filters and groups a dataset of customer reviews to calculate the
average rating for each product category. With ggplot2, they create a bar chart to visualize these
average ratings.

6. What is K-means clustering and how is it used in R?

Answer: K-means clustering is a method for partitioning a dataset into K distinct, non-overlapping
subsets or clusters. It aims to minimize the variance within each cluster. In R, the kmeans() function is
used to perform this clustering.

Example: A marketing team uses K-means clustering to segment customers based on purchasing
behavior. They identify distinct groups, like frequent buyers and occasional shoppers, to tailor
marketing strategies for each segment.

7. How does Linear Regression work in R?

Answer: Linear Regression is a statistical method for modeling the relationship between a dependent
variable and one or more independent variables. In R, the lm() function is used to fit a linear model.

Example: A financial analyst uses Linear Regression to predict stock prices based on historical prices
and trading volume. By fitting a model with lm(), they identify trends and make informed investment
decisions.

8. What is Logistic Regression and when is it used?

Answer: Logistic Regression is used for binary classification problems where the outcome variable is
categorical (e.g., success/failure). It models the probability of a particular class. In R, the glm()
function with a binomial family is used.
Example: A healthcare researcher uses Logistic Regression to predict the likelihood of a patient having
a disease based on factors like age, weight, and blood pressure. This helps in early diagnosis and
treatment planning.

9. What are Decision Trees and how are they implemented in R?

Answer: Decision Trees are a machine learning method used for classification and regression tasks.
They work by splitting the data into subsets based on the value of input features. In R, the rpart
package is commonly used to create Decision Trees.

Example: A loan officer uses a Decision Tree to assess the risk of loan applicants defaulting. By
analyzing features like credit score and income, the tree helps in making approval decisions.

10. What is Time Series Analysis and how is it conducted in R?

Answer: Time Series Analysis involves analyzing data points collected or recorded at specific time
intervals to identify trends, seasonal patterns, and cyclic behavior. In R, the forecast package is widely
used for Time Series Analysis.

Example: An economist uses Time Series Analysis to forecast future unemployment rates based on
historical data. By fitting a model with the forecast package, they can predict and plan for economic
changes.

UNIT-IV

1. What is Hadoop and how did it originate?

Answer: Hadoop is an open-source framework designed for distributed storage and processing of
large datasets using a cluster of computers. It originated from the need to process massive amounts
of data efficiently. The history of Hadoop began with Google’s publication of two papers: the Google
File System (GFS) and MapReduce. These papers inspired Doug Cutting and Mike Cafarella to develop
Hadoop. Named after Cutting's son's toy elephant, Hadoop has become a fundamental tool in big data
processing.

Example: Yahoo! was one of the early adopters of Hadoop, using it to support its search engine and
other data-intensive applications. This demonstrated Hadoop's capability to handle large-scale data
processing tasks efficiently.

2. What is the Hadoop Distributed File System (HDFS) and its components?

Answer: HDFS is the primary storage system used by Hadoop. It is designed to store very large files
reliably and to stream those data sets at high bandwidth to user applications. HDFS has a
master/slave architecture comprising the following components:

 NameNode: Manages the file system namespace and controls access to files by clients.
  DataNodes: Store the actual data and perform read-write operations as directed by the
NameNode.

Example: In a typical Hadoop cluster, the NameNode keeps track of the metadata (e.g., file names,
permissions, and locations), while the DataNodes store the actual data blocks. If a DataNode fails,
HDFS can still retrieve data from other DataNodes, ensuring high availability.

3. How is data analyzed using Hadoop?

Answer: Data is analyzed in Hadoop using the MapReduce programming model. MapReduce divides a
task into small parts and processes them in parallel. The process involves two main steps:

 Map: Processes input data into key-value pairs.

 Reduce: Aggregates the results of the map phase to produce the final output.

Example: Analyzing log files to count the number of times each URL was accessed involves a Map
function that reads each log entry and maps the URL to a count of one, and a Reduce function that
sums these counts for each URL.

4. What does "scaling out" mean in the context of Hadoop?

Answer: Scaling out in Hadoop refers to adding more nodes to a Hadoop cluster to increase its
processing power and storage capacity. This contrasts with scaling up, which involves adding more
resources to an existing node. Hadoop is designed to scale out efficiently by distributing data and
computation across many nodes.

Example: A company starts with a small Hadoop cluster of 10 nodes to process their data. As their
data grows, they scale out by adding more nodes to the cluster, eventually running a 100-node cluster
that can handle significantly larger datasets and more complex analyses.

5. What is Hadoop Streaming and how is it used?

Answer: Hadoop Streaming is a utility that allows users to create and run MapReduce jobs with any
executable or script as the mapper and/or reducer. This is useful for developers who prefer languages
other than Java, such as Python or Ruby.

Example: A data analyst uses a Python script to process log files. With Hadoop Streaming, they can
use their Python script as the mapper and reducer within a Hadoop MapReduce job, leveraging
Hadoop’s distributed computing capabilities without writing Java code.

6. How is the design of HDFS tailored for large-scale data processing?

Answer: HDFS is designed to handle large-scale data processing through:

 Large Block Size: HDFS stores data in large blocks (default 128 MB), which minimizes the
overhead of metadata and improves throughput.
 Replication: Each block is replicated across multiple DataNodes to ensure fault tolerance and
high availability.
  Streaming Data Access: HDFS is optimized for high throughput data access, which is ideal for
batch processing of large datasets.

Example: A video streaming service uses HDFS to store and process user activity logs. The large block
size ensures efficient storage and access, while replication guarantees data availability even if some
nodes fail.

7. How do you develop a MapReduce application?

Answer: Developing a MapReduce application involves writing the Map and Reduce functions,
configuring the job, and submitting it to the Hadoop cluster. Steps include:

 Writing the Map Function: Processes input and produces intermediate key-value pairs.
 Writing the Reduce Function: Aggregates the intermediate data and produces the final output.
 Configuring the Job: Setting input/output paths and specifying mapper and reducer classes.
 Submitting the Job: Running the job on the Hadoop cluster.

Example: A developer writes a Java MapReduce application to count word frequencies in a large text
file. The Map function splits the text into words and emits each word with a count of one. The Reduce
function sums the counts for each word to produce the final word frequencies.

8. How does the MapReduce framework work?

Answer: The MapReduce framework works by splitting the input data into independent chunks
processed by the map tasks in parallel. The framework then sorts the outputs of the maps, which are
input to the reduce tasks. Both the input and the output of the job are stored in a file system.

Example: For processing a log file, the MapReduce framework splits the file into smaller chunks. The
map tasks process each chunk to extract relevant information, and the reduce tasks aggregate this
information to generate the final report.

9. What happens during the shuffle and sort phase in MapReduce?

Answer: The shuffle and sort phase occurs between the map and reduce phases in a MapReduce job.
It involves:

 Shuffling: Transferring the intermediate key-value pairs from the mappers to the reducers.
 Sorting: Sorting the intermediate data by key so that all values associated with a given key are
grouped together.

Example: In a word count job, after the map phase, the intermediate key-value pairs (word, count)
are shuffled so that all counts for each word are sent to the same reducer. The data is then sorted by
word before the reducer aggregates the counts.

10. What are some common MapReduce features and their significance?

Answer: Common MapReduce features include:

  Fault Tolerance: Automatically re-runs failed tasks on another node.
 Scalability: Easily scales to process petabytes of data across thousands of nodes.
 Parallel Processing: Breaks down tasks into smaller sub-tasks that run in parallel.
 Data Locality: Moves computation to where the data resides, reducing network congestion.

Example: A social media company uses MapReduce to analyze user interactions. The framework’s
fault tolerance ensures that even if some nodes fail, the job completes successfully. Scalability allows
the company to process data from millions of users efficiently.

UNIT-V

1. What are Pig and Hive, and how are they used in Big Data applications?

Answer: Pig and Hive are high-level platforms built on top of Hadoop that simplify the process of
querying and analyzing large datasets.

 Pig is a scripting platform that uses a language called Pig Latin. It provides a more
straightforward way to process data using scripts that describe data transformations and
analyses. Pig is ideal for complex data transformations and ETL processes (Extract, Transform,
Load).
 Hive is a data warehouse infrastructure that uses HiveQL (Hive Query Language), a SQL-like
language, to query and analyze large datasets stored in Hadoop. Hive is designed for users
who are familiar with SQL and want to perform data warehousing tasks.

Example: A company needs to analyze customer reviews stored in HDFS. They might use Pig to write a
script that processes and cleans the data, then use Hive to run SQL-like queries to generate summary
reports and insights.

2. What are data processing operators in Pig, and how do they work?

Answer: Pig provides several operators for data processing, each designed for specific tasks:

 LOAD: Loads data from the file system into Pig.

 FILTER: Filters data based on a condition.
 GROUP: Groups data based on a key.
 JOIN: Joins two datasets based on a common key.
 FOREACH: Applies a transformation to each element in a dataset.
 ORDER BY: Orders data based on a specified key.
 DISTINCT: Removes duplicate records.
 LIMIT: Limits the number of records returned.

Example: If you have a dataset of sales transactions and want to find the total sales for each product,
you could use Pig operators to load the data, group it by product, and then compute the total sales for
each group.
3. What are Hive services, and how do they facilitate big data analysis?

Answer: Hive services include various components that facilitate data storage, querying, and
management in a Hadoop environment:

 Hive Metastore: Stores metadata about Hive tables, including schema information.
 HiveServer2: Provides a JDBC/ODBC interface to connect to Hive and run queries.
 Hive CLI (Command Line Interface): Allows users to run HiveQL commands interactively.

These services help in managing large datasets by allowing users to interact with data using HiveQL,
providing tools for data analysis and reporting.

Example: A data analyst might use HiveServer2 to connect to Hive from a BI tool like Tableau, run
queries on large datasets, and generate visualizations without manually handling the underlying data.

4. What is HiveQL, and how is it used for querying data in Hive?

Answer: HiveQL (Hive Query Language) is a SQL-like language used to query and manage data stored
in Hive. It allows users to perform data retrieval, filtering, aggregation, and manipulation using
familiar SQL syntax.

5. What are the fundamentals of HBase, and how does it work with Hadoop?

Answer: HBase is a distributed, scalable, NoSQL database built on top of Hadoop's HDFS. It is designed
to handle large amounts of sparse data across a cluster of machines. HBase provides real-time
read/write access to large datasets.

 Tables: HBase stores data in tables with rows and columns, similar to a traditional database
but with a more flexible schema.
 RegionServers: Manage the data for tables and handle read/write requests.
 HMaster: Oversees the RegionServers and manages cluster metadata.

Example: A company might use HBase to store user activity logs, enabling real-time access and
analysis of data as users interact with their services.

6. What is ZooKeeper, and how does it support HBase?

Answer: ZooKeeper is a distributed coordination service that helps manage and coordinate
distributed applications. It provides services such as configuration management, synchronization, and
naming.

In the context of HBase:

 Coordination: ZooKeeper helps HBase manage distributed resources and keep track of the
state of the cluster.
 Failover: It ensures high availability by coordinating the failover process when a RegionServer
or HMaster fails.
Example: If an HBase RegionServer fails, ZooKeeper helps in transferring the workload to a standby
server, minimizing downtime and ensuring continuous access to the data.

7. What is IBM InfoSphere BigInsights, and what are its key features?

Answer: IBM InfoSphere BigInsights is a comprehensive big data platform that provides tools for
analyzing and managing large volumes of data. It is built on Hadoop and extends its capabilities with
additional features:

 BigInsights Data Explorer: A web-based tool for data exploration and visualization.
 BigInsights Query Workbench: Allows users to run queries and analyze data using SQL.
 Text Analytics: Provides tools for analyzing unstructured data such as text.

Example: A business might use IBM InfoSphere BigInsights to analyze customer feedback from various
sources, gaining insights into customer sentiment and improving their products or services.

8. What are visual data analysis techniques, and why are they important?

Answer: Visual data analysis techniques involve using graphical representations to understand and
interpret complex data. Common techniques include:

 Charts and Graphs: Bar charts, line graphs, pie charts to visualize data trends and distributions.
 Heatmaps: Show data intensity with color gradients.
 Dashboards: Combine multiple visualizations into a single interface for comprehensive analysis.

These techniques are important because they make complex data more accessible and
understandable, enabling users to identify patterns, trends, and insights more easily.

Example: A financial analyst might use a dashboard with line graphs and heatmaps to track and
analyze stock market trends, making it easier to make informed investment decisions.

9. What are interaction techniques in data visualization, and how do they enhance analysis?

Answer: Interaction techniques in data visualization allow users to interact with visualizations to
explore and analyze data more deeply. These techniques include:

 Filtering: Allows users to select subsets of data to focus on.

 Drill-down: Provides the ability to view more detailed data by clicking on specific elements.
 Hovering: Displays additional information when hovering over data points.

These techniques enhance analysis by providing a dynamic and interactive way to explore data,
helping users uncover insights that might not be apparent from static visualizations.

Example: In an interactive sales dashboard, users might filter data by region, drill down into specific
products, and hover over charts to view detailed sales figures, allowing them to perform a more
granular analysis.

10. How do systems and applications support big data analytics, and what are some examples?
Answer: Systems and applications support big data analytics by providing tools and infrastructure for
data storage, processing, and analysis. Examples include:

 Data Warehouses: Central repositories for storing and querying large datasets (e.g., Amazon
Redshift, Google BigQuery).
 Analytics Platforms: Tools for analyzing and visualizing data (e.g., Tableau, Power BI).
 Stream Processing Systems: Handle real-time data processing (e.g., Apache Kafka, Apache
Flink).

These systems and applications help organizations manage and analyze large volumes of data
efficiently, enabling them to make data-driven decisions and gain valuable insights.

Example: A retail company might use a combination of a data warehouse for historical sales data, an
analytics platform for creating visual reports, and a stream processing system to monitor real-time
customer interactions, allowing them to optimize their marketing strategies and improve customer
engagement.

-: THANK YOU RAJNISH :-