DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

(ARTIFICIAL INTELLIGENCE & MACHINE LEARNING)

----------------------------------------------------------------------------------------------------------------

Big Data Analysis

Demonstrate how business problems have been successfully solved faster, cheaper and more effectively
considering NoSQL Google’s MapReduce case study. Also illustrate the business drivers and the findings
on it. Business Drivers:

Google faced a significant challenge with processing and analysing large-scale datasets generated by its search
engine and other services. Traditional relational database systems were struggling to handle the immense volume
of data in a timely and cost-effective manner.

The business problems included:

Scalability: The need to process and analyze massive amounts of data quickly and efficiently. (Velocity)

Cost-effectiveness: Traditional relational databases were proving to be expensive to scale and maintain for such
large datasets.

Performance: The requirement for faster processing to provide real-time insights and results.

Google introduced MapReduce, a programming model, and an associated implementation, which leveraged
NoSQL principles to tackle these challenges effectively. The MapReduce model splits tasks into smaller sub-tasks
that can be executed in parallel across a distributed computing cluster.

Map Phase: The input data is divided into smaller chunks, and a "map" function is applied to each chunk. This
function processes and generates key-value pairs as intermediate outputs.

Shuffle and Sort Phase: The intermediate key-value pairs are sorted and grouped by key across different nodes
in the cluster. This prepares the data for the next phase.

Reduce Phase: The sorted data is passed to a "reduce" function, which aggregates and processes the data for the
final output.

Department of Computer Science & Engineering-(AI&ML) | APSIT

 DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING
(ARTIFICIAL INTELLIGENCE & MACHINE LEARNING)
----------------------------------------------------------------------------------------------------------------

Google's MapReduce demonstrates how NoSQL and distributed computing can successfully solve complex
business problems faster, cheaper, and more effectively. The MapReduce framework enabled Google to process
massive amounts of data, gaining valuable insights and paving the way for advancements in various domains.

Big Data Analysis

Google has leveraged following advantages of MapReduce implementation:

Faster Data Processing: Google's MapReduce framework allowed them to distribute the data processing tasks
across multiple servers, enabling parallel execution. This led to significant speed-ups in data processing. For
instance, tasks that took hours or days with traditional methods could now be completed in minutes or even
seconds.

Cost Savings: The MapReduce approach also resulted in cost savings. By utilizing commodity hardware and
distributing tasks across a cluster, Google could achieve high performance at a fraction of the cost of traditional
solutions. This approach eliminated the need for expensive, specialized hardware.

Scalability: The combination of MapReduce with Bigtable, a distributed NoSQL database, allowed Google to
scale their infrastructure horizontally. As data volumes grew, they could add more servers to the cluster, ensuring
that the system's performance remained consistent even with increasing data loads. predefined schemas,
accommodating the evolving nature of web data.

Efficient Resource Utilization: MapReduce's task distribution ensured optimal utilization of resources. Each
server in the cluster could work on its assigned task, minimizing idle time and maximizing overall efficiency.

Resilience and Fault Tolerance: The distributed nature of MapReduce and Bigtable increased resilience. If a
server failed during processing, tasks could be automatically rerouted to healthy nodes, minimizing downtime and
data loss.

Flexibility: Bigtable's NoSQL design provided flexibility in data modelling. Unlike rigid relational databases,
Bigtable allowed Google to store various types of data without

Department of Computer Science & Engineering-(AI&ML) | APSIT

 DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING
(ARTIFICIAL INTELLIGENCE & MACHINE LEARNING)
----------------------------------------------------------------------------------------------------------------

Following are the business drivers behind Google’s MapReduce discovery,

Volume:

MapReduce is designed to handle massive volumes of data. Traditional data processing systems, like relational
databases, can struggle to scale effectively as data volumes increase. However, MapReduce's distributed
processing model allows it to handle vast amounts of data by dividing it into smaller chunks that can be processed
in parallel across a cluster of servers. This approach ensures that the system can scale horizontally by adding more
servers to the cluster as data volumes grow. This scalability enables efficient processing and analysis of large
datasets without compromising performance.

Velocity:

Velocity refers to the speed at which data is generated and needs to be processed. In the context of real-time or
near-real-time data processing, MapReduce might not be the best fit due to its batch-oriented nature. However,
for scenarios where data doesn't need to be processed in real-time, MapReduce can still be highly effective. By
breaking down data processing into smaller tasks that can be executed in parallel, MapReduce significantly speeds
up the processing time compared to traditional single-threaded approaches. This means that even though
MapReduce doesn't address real-time velocity, it does help handle the high velocity of data by efficiently
processing large volumes of data within reasonable time frames.

Google's MapReduce framework addresses the volume and velocity business drivers. It excels at processing large
volumes of data in a parallel and distributed manner, which leads to efficient data processing and analysis.
Additionally, while not designed for real-time processing, MapReduce can still handle data with a relatively high
velocity within reasonable time frames due to its parallel processing capabilities. These characteristics make
MapReduce a powerful solution for managing and analyzing vast amounts of data efficiently and effectively.

Department of Computer Science & Engineering-(AI&ML) | APSIT