Unit-1: Overview of Big Data

Introduction to Big Data

The quantity of data created by humans is quickly increasing every year as a result of the introduction of new technology, gadgets, and communication channels
such as social networking sites. Big data is a group of enormous datasets that can't be handled with typical computer methods. It is no longer a single technique
or tool; rather, it has evolved into a comprehensive subject including a variety of tools, techniques, and frameworks. Quantities, letters, or symbols on which a
computer performs operations and which can be stored and communicated as electrical signals and recorded on magnetic, optical, or mechanical media.

What is Big Data ?

Big Data is a massive collection of data that continues to increase dramatically over time. It is a data set that is so huge and complicated that no typical
data management technologies can effectively store or process it. Big data is similar to regular data, except it is much larger. Big data analytics is the
use of advanced analytic techniques to very large, heterogeneous data sets, which can contain structured, semi-structured, and unstructured data, as
well as data from many sources and sizes ranging from terabytes to zettabytes.
Big Data refers to large, complex datasets that cannot be efficiently managed, processed, or analyzed using traditional data processing tools. It encompasses the
technology, tools, and practices used to handle vast amounts of data generated from diverse sources, enabling organizations to extract valuable insights.

Characteristics of Big Data (The 5 Vs)

1. Volume:
o Refers to the sheer amount of data generated every second.
o Examples: Social media posts, IoT sensor data, transaction logs.
o The sheer volume of data generated today, from social media feeds, IoT devices, transaction records and more, presents a significant challenge.
Traditional data storage and processing solutions are often inadequate to handle this scale efficiently. Big data technologies and cloud-based storage
solutions enable organizations to store and manage these vast data sets cost-effectively, protecting valuable data from being discarded due to storage
limitations.
PROF. SONALI KHARADE 1
2. Velocity:
o The speed at which data is generated and processed.
o Examples: Real-time data from stock markets or live sports analytics.
o Data is being produced at unprecedented speeds, from real-time social media updates to high-frequency stock trading records. The velocity at
which data flows into organizations requires robust processing capabilities to capture, process and deliver accurate analysis in near real-time.
Stream processing frameworks and in-memory data processing are designed to handle these rapid data streams and balance supply with demand.

3. Variety:
o The diversity of data formats, such as structured (databases), semi-structured (JSON, XML), and unstructured (text, videos, images).
o Today's data comes in many formats, from structured to numeric data in traditional databases to unstructured text, video and images from diverse
sources like social media and video surveillance. This variety demans flexible data management systems to handle and integrate disparate data
types for comprehensive analysis. NoSQL databases, data lakes and schema-on-read technologies provide the necessary flexibility to accommodate
the diverse nature of big data.

4. Veracity:
o The reliability or quality of the data.
o Examples: Handling noisy or incomplete datasets to ensure accurate insights.
o Data reliability and accuracy are critical, as decisions based on inaccurate or incomplete data can lead to negative outcomes. Veracity refers to the
data's trustworthiness, encompassing data quality, noise and anomaly detection issues. Techniques and tools for data cleaning, validation and
verification are integral to ensuring the integrity of big data, enabling organizations to make better decisions based on reliable information.

5. Value:
o The actionable insights or benefits derived from data analysis.
o Example: Personalized recommendations on e-commerce platforms.

PROF. SONALI KHARADE 2

 o Big data analytics aims to extract actionable insights that offer tangible value. This involves turning vast data sets into meaningful information that
can inform strategic decisions, uncover new opportunities and drive innovation. Advanced analytics, machine learning and AI are key to unlocking
the value contained within big data, transforming raw data into strategic assets.

Sources of Big Data

1. Social Media:
o Data from platforms like Facebook, Twitter, Instagram.
o Includes likes, shares, comments, posts, and multimedia.
2. Internet of Things (IoT):
o Data from connected devices like smart thermostats, fitness trackers, and industrial sensors.
3. E-commerce:
o Purchase history, customer reviews, and browsing patterns.
4. Healthcare:
o Electronic Health Records (EHRs), medical imaging, and genetic data.
5. Finance:
o Stock market data, credit card transactions, and fraud detection systems.
6. Telecommunications:
o Call logs, text data, and customer service records.

PROF. SONALI KHARADE 3

 Evolution of Big Data
If we see the last few decades, we can analyse that Big Data technology has gained so much growth. There are a lot of milestones in the evolution of Big
Data which are described below:

1. Data Warehousing:
In the 1990s, data warehousing emerged as a solution to store and analyze large volumes of structured data.
2. Hadoop:
Hadoop was introduced in 2006 by Doug Cutting and Mike Cafarella. Distributed storage medium and large data processing are provided by Hadoop,
and it is an open-source framework.
3. NoSQL Databases:
In 2009, NoSQL databases were introduced, which provide a flexible way to store and retrieve unstructured data.
4. Cloud Computing:
Cloud Computing technology helps companies to store their important data in data centers that are remote, and it saves their infrastructure cost and
maintenance costs.
5. Machine Learning:
Machine Learning algorithms are those algorithms that work on large data, and analysis is done on a huge amount of data to get meaningful insights
from it. This has led to the development of artificial intelligence (AI) applications.
6. Data Streaming:
Data Streaming technology has emerged as a solution to process large volumes of data in real time.
7. Edge Computing:
Edge Computing is a kind of distributed computing paradigm that allows data processing to be done at the edge or the corner of the network, closer to
the source of the data.

Overall, big data technology has come a long way since the early days of data warehousing. The introduction of Hadoop, NoSQL databases, cloud computing,
machine learning, data streaming, and edge computing has revolutionized how we store, process, and analyze large volumes of data. As technology evolves, we
can expect Big Data to play a very important role in various industries.

PROF. SONALI KHARADE 4

Importance of Big Data

1. Business Decision-Making:
o Informed decisions based on data-driven insights.
o Example: Predicting customer churn or optimizing inventory.
2. Personalization:
o Tailored recommendations in e-commerce, entertainment (Netflix, Spotify), and social media.
3. Fraud Detection:
o Real-time analysis of transactions to prevent fraudulent activities.
4. Healthcare Advancements:
o Predicting diseases, improving patient care, and advancing precision medicine.
5. Urban Planning and Smart Cities:
o Traffic management, energy efficiency, and public safety.

Technologies in Big Data

1. Storage:
o Distributed systems like Hadoop Distributed File System (HDFS).
o Cloud storage solutions like AWS S3, Google Cloud Storage, Azure Blob.
2. Processing:
o Batch Processing: Hadoop MapReduce, Apache Spark.
o Real-Time Processing: Apache Kafka, Apache Flink.

PROF. SONALI KHARADE 5

 3. Databases:
o Relational: MySQL, PostgreSQL.
o NoSQL: MongoDB, Cassandra, Redis.
4. Data Visualization:
o Tableau, Power BI.
5. Machine Learning and AI:
o Frameworks like TensorFlow, PyTorch, and Scikit-learn for predictive analytics.

Applications of Big Data

1. Retail:
o Customer behavior analysis, inventory management, and targeted marketing.
2. Banking and Finance:
o Risk management, fraud detection, and algorithmic trading.
3. Healthcare:
o Genomic data analysis, patient monitoring, and drug discovery.
4. Media and Entertainment:
o Content recommendation, audience segmentation, and trend forecasting.
5. Energy Sector:
o Smart grid optimization and renewable energy management.

PROF. SONALI KHARADE 6

Challenges in Big Data

1. Data Privacy and Security:

o Ensuring compliance with regulations like GDPR and HIPAA.
2. Data Integration:
o Combining data from multiple sources with varying formats.
3. Scalability:
o Managing the increasing size and complexity of datasets.
4. Skill Gap:
o Need for skilled professionals in Big Data technologies and analytics.
5. Cost:
o Investment in infrastructure, tools, and expertise.

What is big data analytics?

Big data analytics refers to the systematic processing and analysis of large amounts of data and complex data sets, known as big data, to extract valuable insights.
Big data analytics allows for the uncovering of trends, patterns and correlations in large amounts of raw data to help analysts make data-informed decisions.
This process allows organizations to leverage the exponentially growing data generated from diverse sources, including internet-of-things (IoT) sensors, social
media, financial transactions and smart devices to derive actionable intelligence through advanced analytic techniques.

The benefits of using big data analytics

Ensuring data quality and integrity, integrating disparate data sources, protecting data privacy and security and finding the right talent to analyze and interpret
data can present challenges to organizations looking to leverage their extensive data volumes. What follows are the benefits organizations can realize once
they see success with big data analytics:

PROF. SONALI KHARADE 7

➢ Real-time intelligence
One of the standout advantages of big data analytics is the capacity to provide real-time intelligence. Organizations can analyze vast amounts of data as
it is generated from myriad sources and in various formats. Real-time insight allows businesses to make quick decisions, respond to market changes
instantaneously and identify and act on opportunities as they arise.

➢ Better-informed decisions
With big data analytics, organizations can uncover previously hidden trends, patterns and correlations. A deeper understanding equips leaders and
decision-makers with the information needed to strategize effectively, enhancing business decision-making in supply chain management, e-commerce,
operations and overall strategic direction.

➢ Cost savings
Big data analytics drives cost savings by identifying business process efficiencies and optimizations. Organizations can pinpoint wasteful expenditures
by analyzing large datasets, streamlining operations and enhancing productivity. Moreover, predictive analytics can forecast future trends, allowing
companies to allocate resources more efficiently and avoid costly missteps.

➢ Better customer engagement

Understanding customer needs, behaviors and sentiments is crucial for successful engagement and big data analytics provides the tools to achieve this
understanding. Companies gain insights into consumer preferences and tailor their marketing strategies by analyzing customer data.

➢ Optimized risk management strategies

Big data analytics enhances an organization's ability to manage risk by providing the tools to identify, assess and address threats in real time.
Predictive analytics can foresee potential dangers before they materialize, allowing companies to devise preemptive strategies.

PROF. SONALI KHARADE 8

Careers involving big data analytics
As organizations across industries seek to leverage data to drive decision-making, improve operational efficiencies and enhance customer experiences, the
demand for skilled professionals in big data analytics has surged. Here are some prominent career paths that utilize big data analytics:

➢ Data scientist
Data scientists analyze complex digital data to assist businesses in making decisions. Using their data science training and advanced analytics
technologies, including machine learning and predictive modeling, they uncover hidden insights in data.

➢ Data analyst
Data analysts turn data into information and information into insights. They use statistical techniques to analyze and extract meaningful trends from
data sets, often to inform business strategy and decisions.

➢ Data engineer
Data engineers prepare, process and manage big data infrastructure and tools. They also develop, maintain, test and evaluate data solutions within
organizations, often working with massive datasets to assist in analytics projects.

➢ Machine learning engineer

Machine learning engineers focus on designing and implementing machine learning applications. They develop sophisticated algorithms that learn from
and make predictions on data.

➢ Business intelligence analyst

Business intelligence (BI) analysts help businesses make data-driven decisions by analyzing data to produce actionable insights. They often use BI tools
to convert data into easy-to-understand reports and visualizations for business stakeholders.

PROF. SONALI KHARADE 9

 ➢ Data visualization specialist
These specialists focus on the visual representation of data. They create data visualizations that help end users understand the significance of data by
placing it in a visual context.

➢ Data architect
Data architects design, create, deploy and manage an organization's data architecture. They define how data is stored, consumed, integrated
and managed by different data entities and IT systems.

Future of Big Data

• Edge Computing: Processing data closer to its source for faster analytics.
• AI and ML Integration: Automating insights and enabling predictive analytics.
• Quantum Computing: Addressing the challenges of complex Big Data computations.
• Sustainability: Optimizing Big Data systems to reduce environmental impact.

PROF. SONALI KHARADE 10