Unit 1: Evolution & Introduction to Big Data

1.1 Best Practices for Big Data Analytics

• Data Collection: Ensure data is gathered from diverse and relevant sources.
• Data Quality: Maintain high data quality through cleaning and validation.
• Scalability: Design analytics systems that can scale with increasing data volumes.
• Security & Privacy: Implement robust security measures to protect sensitive data.
• Cost Management: Optimize resources to manage costs effectively.

1.2 Big Data Characteristics

• Volume: The massive amount of data generated and stored.

• Velocity: The speed at which new data is generated and needs to be processed.
• Variety: The different types of data (structured, semi-structured, unstructured).
• Veracity: The uncertainty and reliability of data.
• Value: The potential insights and benefits that can be extracted from data.

1.3 Validating – The Promotion of the Value of Big Data

• ROI (Return on Investment): Demonstrating the financial benefits of Big Data

initiatives.
• Business Impact: Showcasing the positive effects on decision-making and
strategy.
• Case Studies: Presenting real-world examples where Big Data has delivered value.
• Innovation: How Big Data fosters innovation and competitive advantage.

1.4 Big Data Use Cases

• Healthcare: Predictive analytics for patient outcomes.

• Retail: Personalized recommendations and targeted marketing.
• Finance: Fraud detection and risk management.
• Manufacturing: Predictive maintenance and supply chain optimization.
• Telecommunications: Network optimization and customer behavior analysis.
1.5 Characteristics of Big Data Applications

• Scalability: Ability to handle growing data and user demands.

• Distributed Processing: Leveraging multiple machines to process data
simultaneously.
• Real-Time Processing: Analyzing data as it arrives to provide immediate insights.
• Fault Tolerance: Ensuring the system continues to operate despite failures.
• Interoperability: Integration with various data sources and systems.

1.6 Perception and Quantification of Value

• Data-Driven Decision-Making: Using data to guide business strategies.

• Cost Reduction: Identifying inefficiencies and optimizing operations.
• Customer Satisfaction: Enhancing customer experience through insights.
• New Revenue Streams: Creating new products or services based on data analysis.

1.7 Understanding Big Data Storage

• Data Lakes: Centralized storage for raw, unprocessed data in its native format.
• Data Warehouses: Structured storage optimized for query and analysis.
• NoSQL Databases: Non-relational databases designed for high scalability.
• Distributed File Systems: Systems like HDFS that store data across multiple
machines.
• Cloud Storage: Scalable, on-demand storage solutions provided by cloud
platforms.

Unit 2: A General Overview of High-Performance Architecture

2.1 HDFS, MapReduce, and YARN – MapReduce Programming Model

• HDFS (Hadoop Distributed File System):

o Architecture: Master/slave architecture with Name Node and Data Nodes.
o Data Replication: Ensures data availability and fault tolerance.
• MapReduce:
o Map Function: Processes input data and generates key-value pairs.
 o Reduce Function: Aggregates and processes the output of the map phase.
• YARN (Yet Another Resource Negotiator):
o Resource Management: Allocates resources for various applications.
o Job Scheduling: Manages the execution of tasks across the cluster.

2.2 Big Data Overview Analysis of Data at Rest

• Definition: Data that is stored and not actively moving.

• Importance: Crucial for batch processing, historical analysis, and reporting.
• Tools: Hadoop, Hive, Pig, and HBase for processing and analyzing data at rest.

2.3 Hadoop Analytics: Limitations of Existing Distributed Systems

• Scalability Issues: Traditional systems struggle with the volume and velocity of Big
Data.
• High Costs: Maintaining and scaling traditional systems can be expensive.
• Inflexibility: Rigid architectures are not well-suited for unstructured data.

2.4 Hadoop Approach and Architecture

• Distributed Computing: Breaks down tasks and distributes them across multiple
nodes.
• Fault Tolerance: Automatically replicates data and reruns failed tasks.
• Data Locality: Processes data where it is stored to minimize data movement.

2.5 Distributed File System: HDFS and GPFS

• HDFS (Hadoop Distributed File System):

o Data Blocks: Stores large files by splitting them into smaller blocks.
o Replication: Each block is replicated across multiple nodes.
• GPFS (General Parallel File System):
o High Performance: Optimized for large-scale, high-performance computing
environments.
o Flexibility: Supports various file types and workloads.

2.6 Internals of Hadoop MR Engine

• Job Tracker: Manages the MapReduce jobs, scheduling, and monitoring tasks.
• Task Tracker: Executes the individual map and reduce tasks on Data Nodes.
 • Data Shuffling: Transfers intermediate data between the map and reduce phases.

2.7 Hadoop Cluster Components

• Name Node: Manages metadata and namespace for HDFS.

• Data Node: Stores actual data blocks and serves read/write requests.
• Resource Manager: Oversees resource allocation for applications in the cluster.
• Node Manager: Manages resources on individual nodes.

2.8 Hadoop Ecosystem

• Hive: Data warehouse infrastructure for Hadoop, supports SQL-like queries.

• Pig: High-level scripting language for data transformation.
• HBase: Distributed, scalable NoSQL database on top of HDFS.
• Oozie: Workflow scheduler for managing Hadoop jobs.
• Zookeeper: Coordination service for distributed applications.

2.9 Evaluation Criteria for Distributed MapReduce Runtimes

• Performance: Speed and efficiency of processing large datasets.

• Scalability: Ability to handle increasing data and processing demands.
• Fault Tolerance: Resilience to node failures and data loss.
• Ease of Use: Accessibility and simplicity of the runtime environment.

2.10 Enterprise-Grade Hadoop Deployment

• Security: Implementing authentication, authorization, and encryption.

• Data Governance: Policies and procedures for managing data lifecycle.
• Compliance: Adhering to industry standards and regulations.
• Integration: Seamless integration with existing enterprise systems.

2.11 Hadoop Implementation

• Cluster Setup: Configuring hardware, installing Hadoop, and setting up nodes.

• Data Ingestion: Importing data into HDFS using tools like Sqoop and Flume.
• Job Execution: Running MapReduce jobs and monitoring performance.
• Optimization: Tuning parameters for better performance and resource utilization.
Unit 3: Advanced Analytical Theory and Methods

3.1 Overview of Clustering – K-Means, Use Cases

• K-Means Clustering:
o Algorithm: Partitions data into K clusters by minimizing the variance within
each cluster.
o Use Cases: Customer segmentation, image compression, anomaly
detection.

3.2 Overview of the Method

• Initialization: Choose initial centroids for K clusters.

• Assignment: Assign each data point to the nearest centroid.
• Update: Recalculate centroids based on the assigned points.
• Convergence: Repeat the process until centroids no longer change.

3.3 Determining the Number of Clusters

• Elbow Method: Plotting the sum of squared errors to find the optimal number of
clusters.
• Silhouette Score: Measures how similar a point is to its cluster compared to other
clusters.

3.4 Clustering, Classification, Segmentation

• Clustering: Grouping similar data points together.

• Classification: Assigning data points to predefined classes.
• Segmentation: Dividing data into meaningful, actionable subsets.

3.5 Linear Regression

• Definition: A method for modeling the relationship between a dependent variable

and one or more independent variables.
• Applications: Predicting trends, pricing strategies, and risk assessment.

3.6 ML Search: Indexing and Indexing Techniques

• Indexing: Creating data structures that improve the speed of data retrieval.
 • Techniques: B-trees, hash tables, inverted indexes for efficient search operations.

3.7 Create Inverted Index Using JAQL

• JAQL: A JSON-based query language for analyzing large datasets.

• Inverted Index: A data structure that maps terms to their locations in a dataset,
useful for search engines.

3.8 Data Explorer Bundling Hadoop Job: Application, Diagnostics

• Data Explorer: A tool for visually exploring and analyzing data.

• Hadoop Job Bundling: Packaging multiple Hadoop jobs together for execution.
• Diagnostics: Monitoring job performance and identifying bottlenecks.

3.9 Reasons to Choose and Cautions

• Reasons to Choose:
o Scalability: Ability to handle large datasets.
o Cost-Effectiveness: Open-source tools reduce costs.
• Cautions:
o Complexity: Steep learning curve for setting up and managing Hadoop.
o Data Quality: Poor data can lead to inaccurate analysis.

3.10 Classification: Decision Trees

• Definition: A tree-like model used to make decisions by splitting data based on

feature values.
• Applications: Credit scoring, medical diagnosis, marketing strategies.

3.11 Overview of a Decision Tree

• Nodes: Represent features in the data.

• Edges: Represent decision rules.
• Leaves: Represent the final decision or outcome.

3.12 The General Algorithm – Decision Tree Algorithms

• ID3: Uses information gain to select the feature that best splits the data.
• C4.5: An extension of ID3 that handles both categorical and continuous data.
 • CART (Classification and Regression Trees): Builds binary trees for both
classification and regression tasks.

3.13 Evaluating a Decision Tree

• Accuracy: The proportion of correctly predicted instances.

• Precision & Recall: Measures of the relevance and completeness of the
predictions.
• Confusion Matrix: A table that compares actual and predicted classifications.

Unit 4: Real-Time Analytics

4.1 Introduction to Streams Computing

• Definition: Real-time processing of data streams as they arrive.

• Applications: Fraud detection, social media analysis, IoT data processing.

4.2 Challenges/Limitations of Conventional Systems

• Latency: Traditional batch processing introduces delays.

• Scalability: Difficulty in scaling to handle high-velocity data streams.
• Complexity: Managing and processing streaming data requires specialized skills.

4.3 Solving a Real-Time Analytics Problem Using Conventional Systems

• Limitations: High latency and inability to provide instant insights.

• Workarounds: Using mini-batch processing and approximations.

4.4 Challenges to be Solved - Scalability, Thread Pooling, etc.

• Scalability: Ensuring the system can handle increasing data volume.

• Thread Pooling: Efficiently managing system resources for concurrent processing.
• Fault Tolerance: Maintaining system reliability despite failures.
4.5 Understanding the Challenges in Handling Streaming Data from the Real World
and How to Address Those Using Stream Computing

• Data Quality: Dealing with noisy, incomplete, or inconsistent data.

• Real-Time Processing: Ensuring low-latency data processing and analysis.
• System Integration: Integrating streaming systems with existing data
infrastructure.

4.6 Benefits of Stream Computing in Big Data World

• Immediate Insights: Real-time data processing provides instant feedback.

• Scalability: Stream computing systems can scale with data velocity and volume.
• Cost Efficiency: Reduces the need for extensive storage by processing data on the
fly.

4.7 Real-Time Analytics Platform (RTAP)

• Definition: A platform designed to process and analyze data streams in real-time.

• Components: Data ingestion, stream processing engine, analytics dashboard.

4.8 Real-Time Sentiment Analysis

• Definition: Analyzing social media and other sources to gauge public sentiment in
real-time.
• Applications: Brand monitoring, customer feedback analysis, market research.
• Tools: Apache Storm, Spark Streaming, Twitter API.